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THE. GRE COOPERATIVE VikLIDlTY STUDIES PROJECT 



Kenneth M. Wilson 
Educational Teating Service * 

Decisions to admit some applicants -for graduate study and to 
reject others have serious Implications for Individuals, graduate 
schools, and society. It Is of the utmost Importance that 4II • 
such decisions should be guided by up-to-date and reliable, knowledge 
regarding the predictive validity of data employed in screening*) 
applicants for admission. All parties to the development Of the 
Graduate Record Examinations (GRE) Program, from the outset, have 
recognized the need for empirical evidence regarding the predictive 
validity of GRE tests and other preadmissions variables. 

Concern for predictive validity has been expressed directly 
in the form of GRE-Board support for a variety of ad hoc projects 
that have had the general aim of helping to increase the amount and 
quality of Information available about the validity of GRE Aptitude 
and Advanced Tests, and to improve the validity study process 
generally* 

Projects undertaken by ETS, at the suggestion of either the 
Committee on Testing of the Association of Graduate Schools (AGS) 
or the Graduate Record Examinations Board, have been of three 
principal types;* 

1. those concerned with collecting and disseminating informa- 
tion and Insights gained from locally conducted institu- 
tional/departmental validity studies (e.g., Lannholm and 
Schrader, 1951; Lannholm, 1960, 1968, 1972); 

2. those concerned with conducting centrally planned validity 
studies with the cooperation of selected graduate schools 
and/or departments (e.g., Lannholm, Marco, & Schrader, 
1968; Boldt, 1975); and 

3. those concerned with the study of particular applied, 
methodological, or conceptual aspects of the validation 
process or with particular problems and issues (e.g., 
Boldt, 1975; Carlson, Evans, & Kuykendall, 1973; Reilly, 
1971, 1974; Rock & Harmon, 1972). 

These diverse approaches have sought to encourage and improve 
validation research by Improving the scope and quality of informa- 
tion available to graduate schools regarding the validity of GRE 



* A detailed review of representative projects is provided in 
Appendix A. 



t«8t8 and the different way« in which validity studifea might be.^ 
carried out; by actually conducting^atudl^ uajj^ dajlp provided by* 
selected institutions or departments; ^nd by iocusing attention 
on important validation research problems and exploring promising 
developments tof coping with these problems # 

Despite the impetus re)>resented by these projects^ spanning a , 
full quarter of a century^ tfte number of graduate level validity 
studies has remained lov# The Cooperative Validity Studies ProJ^^t 
described in this report^ reflects an extension and intensification 
of efforts on the part of the Graduate Record Examinations Board to 
promote and facilitate the participation of graduate* sohools and 
departments in the GRE valldity«*study process* 

An immediate objective of the proje^'dt was to enlist the co* 
operation* or graduate schools and departments* in studies designed 
to obtain up-to-date information regarding the j)r edict iye validity 
of GRE Aptitude and Advanced Tests' and other variables used in 
admission (such as the Undergraduate Grade Point Average or UP6A) 
with respect to relevant criteria of perf rmance in graduate study* 
It vas assumed that experience gained ^n a.veloping and implementing 
cooperative studies with ^ wide range of graduate schools and depart- 
ments would contribute to the development ot longer'^Xerm arrangements 
through which the GRE Program might facilitate the recurring 
participation of schools and departments in GRE validation research* 

This report provides an overview of the Cooperative Validity 
Studies Project* The principal project activitiest f indingSt and 
conclusions arc described following a brief lamination of some 
of the reasons for (a) the comparatively low level of validity study 
actlvit> at the graduate level and (b) the need for cooperative 
interaction between graduate schools and departments , the GRE 
Boardt and the GRE Program at ETS in validation research* 
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.The volume, scope, arid coverage of valldl-ty st]idy activity in 
graduate school settings has been quite low both in relatj^pn to 
the number of ^ettln^a.ln which validity studies could and should be 
6onducted (e*g#,' all graduate departments In which GRE scores may 
affec^ admissions decisions) and »as compared with the Volume of 
validity study activity In other educational settings ^here clrcum*- 
stances have been more conducive to the widespread applic^lon ^^^9 
standard validity study models 2tnd procedures# / \ * " 

In undergraduate and^ law school settings, for example, the- 
validity study process has been faciliate^ by (a) the existence of / 
comparatively large entering cohdrts of studetfts engaged in compar- 
able academic pursuits, especially during tne initial phases of 
their educational programs, and (b) the ^^^eneral acceptance of one 
performance index--namely, the first-year grade point avera^e~as 
the criterion against which to validate* admissions variables* 
Questions regarding the predictive validity of individual admissions 
variables and the most effective combinations of those variables for 
predicting flrstiyear grades are addressed syistematically by applying 
i^tandard statistical models, principally multiple regression analysis, 
to data for sizeable samples typically corresponding to an entering « 
first-year class for each institution (Schrader, 1971, 1977)* 

These conditions have l^en conducive to the reguiariz^tioi} 
of institutional participation in validity studies employing 
standard design and methodology* Summaries of findings, prepared 
from time to time, provide normative perspective with respect to 
trends^ across institutions and over time in patterns of correla-* 

tional validity for relevant predictors* 

• . • 

o In a recent review of law school validity studies, for 

example, Schrader (1977) drew upon the results of over 625 
studies involving the Law School Admission Test and an 
undergraduate grade point average, in relation to a 
first-year grade average criterion, completed for 150 law 
schools between 1948 and 1975* 
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The number of validity studies In undergraduate settings is 
also high* During the period 1964-1968, for example, almost 
1,900 validity studies involving College Board tests in 
relation to freshman-level grade point average, criteria were 
completed at ETS alone ( Internal communication)* 

In graduate settings the situation is much more complex, and 
validation research has not become part of an established routine* 
Each graduate scnool has several ••entering classes" each year, 
corresponding to distinctive subgroups definable, for example, in 
terms of field of study (department in whicL enrolled), type of 
degree program (e*g*, terminal m;ister^.s, master^s-d<H*ti)ratc* sequence. 



docttfrai only), and educational etatua (e.g., :flr8t-tiinp' graduate 
'student, master'a-degree holder, etc.)* These auhgrou^s represefit 
cohorts for which separate validity studies are Itksfly^o be needed, 
and 4ach such cohort typically Is qul^e snallf J 

' • - . .' . ' . . . ' 

Problems related to small *«ani>l.es and therefore unstable 
e3tlinates of parameters (underlying, Jfelationshlps between predictor 
and criterion' variables), endemic In graduate school validation . 
research, have been compounded by lack of' a working consensus 
regarding the most uBe|ul (apgroprlate or relevant) criterion of 
"success" in graddat'eTatudy (Wllllnghami 197A). The graduate grad^ 
polnt' average fceflects one relevant dimension of performance— and' It < 
has been the most consistently employed criterion In ad hoc validity 
studies— but It has failed to command the widespread acceptance 
adcorded the first-year grade point average a» a criterion In 
undergraduate and professional school settlrigs* 

The foregoing litany of deterrents helps 'to explailn* the 
fact that comparatively few graduate schools assess *he» validity of 
GRE tests, or other ^miaftianft vaiMables, systfematlcally and regularly 
According to a GR£-sp^sored. sWey of member institutions of the 
Council of <5rad»jate SctWoft (BuW, 19.70), 57 percent of 245 respon- , 
dents Indicated no valldlt^ 'studHrttl^lty within the most recent 
three-year period, 18 percent reported.^only limited stupes (i.e., 
studies involving only a few departments or programs), and 11 percent 
reported "ui^kfiown" in response to the question, about validity studies. 

. To be surd, a^hoc validity studies' involving GRE scores in 
relation to'varloas measures of student performance or success in 
graduate study have been conducted from time to time in a variety 
of graduate school, departmental, and/or disciplinary samples. * 
However, in his review of GRE valldatiow research covering a 20- 
year perlbd, . 1952-1972, Wllllngham "(1973, 197A) could draw upon 
the results of only 43 studies InvplyingNcorrelatlons of . GRE 
Aptitudfe or Advanced Test scores and jindergraduate* grade point 
avera^ (UGPA) with diverse criteria qf "success," principally 
the graduate grade point average, but including faculty ratings, 
departmental examinations, Ph.D. attainment versus nonattainment, * 
and tiine taken to attain the degree. 

» 

These ad hoc studies provided evidence that GRE ^scores and UGPA 
were positively related to each of a number of different perform- 
ance criteria in samples, typically corresponding to graduate 
departments, from a variety of disciplines. At the same time, most 
of the studies reviewed were conducted during the 1950'8 and 1960 b, 
leaving unresolved important questions regarding the correlational 
validity of these predictors in more recently enrolled cohorts of 
graduate students. Also, significant questions regarding the 
predictive validity of GRE scores and undergraduate grades In 
various subgroups— for example, women, minorities, older students, 
or foreign students—could not be addressed on the basis of findings 
of the ad hoc studies reviewed by Willingham. 



A nnnperattve Studlea Rationale 



Gradiiate schools share with all uthar educational institutioni* 
a continuing need for and a' responsibility to develop current 
answers to questions regarding the predictive validity of standard 
tests aid other var.iables used in screening applicants for admi^lon. 
These are recurring questions toVhlch schools need up-to-date 
answers if they are to keep up with changes in student populations, 
graduate-program characteristics, conditions of test itse, as well as 
changes dn the nature* of admissions variables per de. To answer 
these questions, empirical evidence is heeded regarding the relation- 
ship of admissions variables to clearly defined and relevant, if 
less than ultimate, performance criteria in representative cohorts, 
demographic subgroups, and admissions contexts. 

Given the complexities of conducting graduate level validity 
studies, it seems unlikely that concerned graduate schools will be 
able to monitor ORE predictive validity systematically and thoroughly 
if chey are forced to rely solely on self-initiated validity studies 
that are based on small departmental sami^les and are lacking in 
comparability of design, methodology, sample or cohort definition, 
and the like* 

The Graduate Record Examinations Board in commissioning the 
Cooperative Validity Studies Project premised its action on the 
assumption that improvement of GRE validation research is most 
. likely to result from sustained cooperative interaction betweexi 
all concerned parties: graduate schools and departments, ETS 
staff,,, and the Graduate Record Examinations Board. 

Cooperative interaction, of course, may take a variety of fcrms. 
The Cooperative Validity Studies Project was undertaken to develop 
and test the effectiveness of one or more validity study models as a 
basis for implementing GRE validity studies in cooperation with 
concerned graduate schools and departments. 



PART II. DEVELOPING AND IMPLEMENTING COOPERATIVE STUDIES , 



As Indicated above> thj^ Cooperative Studies project was 
initiated in the fall of 1975^ with the general aim of developing 
and testing, models and procedures for facilitating the participation 
of graduate schools and departments in cooperative 6RE validity 
studies* An immediate objective of the project was to generate 
up-to-date empirical evidence regarding the correlational validity 
of GRE Aptitude and Advanced Tests^.as well as other preadmissions 
variables^ with respect to relevant criteria of performance in 
graduate study in clearly defined student cohorts in a variety of 
departmeh^L settings*^ It was assumed that experience gained during 
the course^of the project would contribute to the development of 
longer-term aVrangements for regularizing the participation of 
graQUeea schools and departments in GRE validity studies* 

At the outset > several types of arrangements and/or models 
were envisaged as potentially usefufl for developing and implementing 
cooperative validity studies* For a variety of rea8on8» it was 
decided that the most promising approach would be for ETS to develop 
a structured validity study model, specify the data needed to carry 
out studies In accordance with the model, and ofcfer to conduct 
studies and prepare report^ for all institutions and departments 
willing and able to provide the needed data* 

It was reasoned that by offering to the graduate school community 
a sharply focused validity study model with limited data requirements 
and relatively few conditions for participation, individual graduate 
schools and/or departments would be able rather quickly (a) to 
assess the relevance of the model to their interests and circumstances 
and, if interested, (b) to indicate their readiness to participate 
in and provide the data required to complete studies* Findings of ^ 
studies conducted using this approach, embodying standard data and 
study design, would be comparable across institutions and departments, 
and would permit the comparison of findings and the assessment of 
trends between and across fields of study* 

Recruitment of participants was Initiated In April, 1976, through 
a survey of graduate deans of Institutions comprising the membership 
of the Council of Graduate Schools (COS). The survey was designed 
primarily to Ident^lfy prospective participants In cooperative validity 
studies* However,- It was also concerned (a) with ascertaining the 
types of questions or issues about GRi: validity that were current on 
campus, aod (b) with assessing the current status of validation 
research in CGS schools, especially the extent and nature of validity 
study activity since 1970* 

With a covering letter from the Chairman of the (Jraduate Record 
Kxaminatlons Board, the survey was mailed in April, 19/^>, la 




graduate deans.* A total of 244 deans or their representatives 
replied. Some degree of interest in the possibility of participating 
in cooperative studies was indicated by 130 of the respondents. The 
fact that so many deans reported some aegree of departmental interest 
in the possibility of participating in validation research may be 
understood best when considered in relation to the extremely low 
incidence and uneven nature of locally conducte'd validity study 
activity reported. Only 34 respondents indicated that studies 
involving either the GRR Aptitude Test only, {or both the Aptitude 
and Advanced Tests, had been completed since 1970; 30 schools 
reported limited studies in progress.** 

Survey respondents were aslced to specify departmental or 
progralh areas in which there was an active interest in the validation 
or further /validation of GRE tests as predictors yf student performance. 
It was understood that designation of an area as actively interested 
would not involve a commitment, but only an indication of readiness . 
to explore actively the possibility of participating in studies, 
given mutually acceptable models and procedures. As indicated 
above, 130 respondents (53 percent) indicated some interest in 
cooperative studies at institutional or departmental levels. 

As the next step in the process of recruiting participants, all 
interested respondents were asked to review and assess the relevance 
to their interests and circumstances of a short-term validity study 
model. The salient features of the model are briefly described 
below.*** 

o The model focused on the performance of first-time, 
full-time graduate students who entering a degree 
program In the fall of 1974, and the fall of 1975. 
Two entering cohorts were specified in order to 
augment saq^ple size. 

o Departments enrolling 25 or more such students in the 
two cohorts combined were encouraged to participate 
by providing In roster format a very limited set of 
data on each student. 



f 

* A copy of the covering letter and the survey forms used are included 
in Appendix B-1 . 

** A detailed report of survey findings related to the current status 
of local GRH validity study activity and related Issues and concerns. 
Is provided In Appendix B-2. Results are largely consistent with those 
reported by Burns (19 70) which Indicated a low volume of validity study 
activity prior to 19 70. 

*** A detailed descrlpMon of the one-year study model is provided in 
Appendix B-i; a spi'ciinen act of data-collect inn materials is also Includ 



o More specifically* departments were asked to provide 
scaled scores on the 6RB Aptitude Test (Verbal and 
Quantitative)* and/or the 6RE Advanced Test scores, 
plus at least one measure of performance (or criterion 
measure) during the first year of study (e.g., first-year 
Graduate Grade Point Average or GPA» faculty ratings, 
end-of«*year examinations, etc* Departments ifere encoiiiraged 
to provide of Oin Undergraduate GPA* 

o No limit was set on the nuniber of departments for which 
studies would be mad^» nor was there any emphasis on 
specific departn^nts or types of programs* As indicated 
above, however, the model did involve the explicit dellmi-* 
" tat ion that the samples were to Include only f irst-»tlme 
graduate students (anywhere) # This limitation was included 
to provide a very necessary measure of control over 
educational status at point of entry into a program# 

o Departments were asked to j^dentify **foreign students for 
whom English is not the native language*' if they were 
included in a sample, since lack of fluency in English may 
affect performance on GRE tests* Coding for sex and 
ethnicity was optional* 

o Interested departments could also provide data on other ^ 
variables of interest***»e*ge, age at entry, qxiality of 
undergraduate school, date of bachelor^s degree, etc* 

The basic approach was designed to encourage graduate schools 
and departments to participate in cooperative studies by minimizing 
the strictures and requirements related to data collection* It 
was understood that ETS would analyze data and prepare a report of 
findings for each graduate school, without cost to the participants* 

During the period April, 1976, through October, 1976, 44* 
graduate schools indicated an intention to provide data on one 
or mote departmental samples after reviewing the one«*year study 
model proposed* A total of 35 schools ultimately provided data 
for one«-year studies involving from one to seventeen departments 
per school* 

Several graduate schools with established arrangei^nts for 
sharing data on admissions^-related questions expressed an interest 
in participating in a study» based on a two-year models involving 
the collection of data on first«-tlme, full-'tlme students entering 



* These studies were initiated prior to the introduction of the 
restructured Aptitude Test that yields an Analytical Ability (or 
GRE«»A) score In addition to the Verbal and Quantitative scores* 
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selected departments in the fall of 1974. Pour of these schools 
ultimately provided data for studies designed to assess the predic- 
tive validity of GRE scores using two-year cumulative CPA or other 
criteria** 

Thus» 39 graduate schools were recruited as participants in GRE 
^ validity studies* These schools are listed in Table 1* 



* Appendix B-A provides a brief description of the special studies 
undertaken In cooperation with these schools. Results of the two-year 
studies were generally comparable with those that will be described 
In the subsequent section for the basic one year studies. However, 
because of differences in definitions and design, results for the 
two-year studies are not Included In the sumiaarizations that 
are provided In Part III. 
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Table 1 



Graduate^Schools Participating in 
/cooperative Studies 



School 



Air Force Institute of Technology 
Auburn University 
Baylor University 
Bradley University 
Brovm University 

California State University at Fullerton 

Florida Technological University 

Fort Hays Kansas State College 

Harvard University i 

Hofstra University 

Indiana University at Bloomington 

Louisiana State University. 

Loyola University at Chicago 

The Ohio State University 

Old Dominion University 

CXregori State University 

Princeton University 

Stanford University 

State University of New York at Stony Brook 
University of Arizona at Tucson 
University of California at Berkeley 
University of Colorado at Boulder 
University of Hawaii at Honolulu 
University of Illinois 
University of Kentucky 
University of Mcissachusetts 
University of Miami 
University of Michigan ^ 
University of Missouri at Rolla 
Univei;sity of Montana 
University Df New Orleans 

University of North Oar4)liua at Chapel Hill 

University of N4>tre Dame 

University of Oklahoma 

UuiversHy of Texas at Arlinj^'lon 

University of Virginia 

University of Wisconsin 

Virginia State (U)iLege 

Washington State University at Pullman 

Wayne State University 
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Part III. GENERAL CHARACTERISTICS AMD P.lINCIPAL FIKDINGS OF 

THE CODPERATIVE .STUDIES ^ 

During the course of the project, validity studies based 
on the standard one-year model were completed for 35 graduate 
schools, and studiesvbased on a special two-year model were completed 
for four schools. Data were available for from one to 17 departments , 
per school. For each school, data were analyzed by department, and 
findings were summarized In an "Institutional report. One institu- 
tional report (without identification) is included in Appendix C to 
illus^ate the nature, scope, and limitations of validity studies 
involving small departmental samples.* 

The institutional report provided a basis for organizing and 
summarizing validity study findings in such a way as to be of 
greatest direct interest to eacl' participating graduate school. 
However, since the departmental sample was the basic unit of analysis 
in all studies, it is more meaningful to examine the general character- 
istics of samples and data and to summarize findings for departments 
grouped by field of study or discipline than to do so by institution. 
Accordingly, in this section, characteristics of the departmental 
samples, the data employed in the standard one-year studies, and the 
principal findings of the studies are summarized l^y field or 
discipline. 



ERIC 



General Characteristics of Sa mples and Data 

The standard one-year studies were designed to assess the 
relationship of GRE and other predictors, as available, to one or 
more measures of student performance during the first yeat of 
graduate study, in departmental samples from a clearly delimited 
population, namely, first-time graduate students (anywhere), who 
were classified as full-time and enrolled in a degree program. 
Limited data were requested for cohorts entering in the fall of iy/4 
and the fall of 1975, combined to augment sample size. 

The 35 scnools -participating in one-year studies provided 
data for over 130 departmental samples meeting study definitions 
from a wide range of fields or disciplines. The first-yeaj: Graduate 
Grade Point Average (Grad GPA) was provided as the criterion or 
performance measure for essentially all the samples. Other measures 
such as faculty ratings or grades in critical courses or course 
sequences were infrequently provided. Scores on the GRE Aptitude 
Test (Verbal or GRE-V, and Quantitative or GRE-Q) were also common 
to all samples. In some samples Aptitude scores were supplemented 



See tho report for School A» Appendix C-I. 
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by an Undergraduate GPA (seldom fully described but characteristically 
on a scale ranging from A •> *A thprough F •* 0) and/or scores on one 
of the 20 6RE Advanced Tests offered by the GKE Program. > 

The departioental samples were typically quite small* Many 
Included fever than 25* cases and were characterised by missing data 
patterns«-«-observatlons or scores oq one or more variables were 
missing for one or more students In most samples* 

Some samples Included- only pr^^opectlve master^s c^ndldates> 
some only prospective doctoral' candidates, and others Included 
both* In e%sentlally all samples that Included both prospective 
master^s and |irospectlve doctoral students, flrst«-year departmental 
programs and evaluation procedures were reportedly comparable for 
both** 

Table 2 shows for each of 19 fields or clusters of fields 
(a) the number of departmental samples with data on the respective 
predictors (GRE Verbal, GRE«-quantltatlve» GRE-Advanced, and/or 
Undergraduate GPA) and (b) the average (mean) number of students per 
department with observations on a given predictor* For example. It 
may be seen that the 22 Bioscience samples had observations on 
GRE-Aptltude (GRE**Verbal and ^-Quantitative), but that only 12 of 
these samples had 25 or more students with Aptitude scores; the mean 
number' of 'students with Aptitude scores was 26*4* Similarly, 
only 13 Bioscience samples Included observations on the Biology 
Advanced Test (for an average of 16*8 students) ^ereas 14 Included 
an^ Undergraduate GPA, etc* 

Several characteristics of the samples available for analysis 
are clearly discernible in Table 2, Including the following: 

o As previously noted, GRE Aptitude scores^-were available for 
all samples* However, only about half (70 of 138) of the 
samples included scores on a GRE^Advanced Test ; about 
38 percent of the samples (80 of 138) provided an 
Undergraduate GPA* 

o The characteristically small size of the departmental 
samples, which it will be recalled Included students 
in two entering cohorts (fall 1974 and fall 1975, 
combined), is pointed up clearly in the table* Only 
86 of 138 samples included at least '25 students with GRE 
Aptitude Test scores ; only 28 of 70 samples provided data 
on GRE Advanced Tests for as many as 25 students, and 28 
percent of the samples included fewer than 25 students 
with UGPA as a predictor* 



* Most of the general features described above are illustrated in 
the study for School A, Appendix C-1* 
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Table 2 

KumUcr of Samples with ObBervatlotti* on the 



»» 

• 


Respective 


I'riJictora and Muau Sanplc Size P«r 


Predictor 














MiiinKor of snmnltiii 


Mean sample alzv 










CRE- 


UCPA 


GRE- 


GRE- 


UGPA 


Department 




Aptitude 






Apc i tudt! 










22 


(12)* 


13 (2) 


14 (5) 


26.4 


16.8 


29*9 


Chomis t ry 




12 


( o) 


7 (5) 


8 (7) 


32.4 


31.3 


46.2 






10 


( 4) 


4 (0) 


b (2) 


20.2 


10.8 


18.2 






6 


( 3) 


2 (0) 


2 (0) 


25.7 


17.0 


16.0 


PhynicH 




5 


( 3) 


4 (3) 


2 (2) 


36.6** 


44.0** 


75.5** 


Geology; Geophysics 




5 


( I) 


A (1) 


1 (1) 


19.6 


IS. 8 


. 39.0 


Kcunomlcs 




6.( A) 


3 (3) 


3 (3) 


34.0 


36.7 


41.7 


Aiithropolo>iy 




3 


( 2) 




1 (I) . 


!ll.7 


29.5 


47.0 


Education^ 




7 


( 6> 


2 (2) 


5 (5) 


41.7 


66.4 






6 


( 3) 


5 (0)' 


A (2) 


31.7 


24.4 


36.0 


History 




10 


(10) 


7 (3) 


b (») 


34.8 


22.8 


35.5 






A 


( 4) 


2 (1) 


3 (3) 


45.3 


37.5 


52.3 


Psychology 




12 


(10) 


7 (b) 


7 (4) 


43.4 


40.0 


43.7 


SDclolo>;y^ 




7 


( b) 


J (1) 


5 (4) 


41. 0** 


14.3 


29.3 


Library Science?* 




3 


( 3) 




3 (3) 


39.0 




39.3 


Fine Arts-* 




6 


( 6) 




5 (5) 


40.7 




43.8 


Music 




4 


( 4) 


3 (2) 


1 (1) 


44.5 


26.3 


33.0 


Philosophy 




5 


( 0) 


2 (0) 


2 (0) 


16.4 


8.5 


20.0 


Laii>;ua>;os^* 




5 


( 1) 


2 (0) 


2 (0) 


17.6 


16.5 


14.0 


Total 




138 


(Sh) 


70 (28) 


81 (5b) 


3J.1 


25.0 


38.0 


*N!imbers In p.ironthfses 


imlloattf 


the 


iiumbor of samples for which N " 


25 or greater • 






**MtMn Inf latt ij !>> one 


re 


Lit ivc'ly 


larn 


f dcii.irinifnt.il 


tiamp 1 (> . 








•^IncluUes Oct'. iiu*>»,r.iphy. 


."..n iiu' Knvl r<' 


tiiiR'itt.il Sclcnoo 


, All led Hoalth ' 








I lie 1 udcs KnK i necr in>: ami F.ic i 1 1 1 ie:i 












Includes C«)mptitt»r :te 


Imcoii, Appl led 


Math and Stalls 


t ICH 








*hnt'lude« Vvu:.uii>:ul 




1 Adult KiluiMllon, Kdiu-.U Aamlnlsir.it Ion 









'"tncludcs Tubli. Adminl.itrati-iti 

'liKliidfH Social W..ik. ITbait I'laniiliiK, I'ubll.- l'i)l).y tiltutU's 

«lncludt-: St ch aii.l IMu-atrt. l)ram.4 and Comimmu aliou. Speotrh and (-..ttumiuicat loti and .lotttnal ism 

•'Includes two Hl:,|.a«ii . Ornuialc, u.u- Ftfuch and ..m- undi t U-r»-.it lat .d for.-lKU l.annuaKos at..; l.i loraturca 
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o Mean sample slze» In one or two cases (particularly 
physics and sociology) Inflated by the presence of one 
atyplcally large samplet vas approximately 32 for the GRE 
Aptitude analyses^ 25 for GRE Advanced Test analyses, and 
38 for U6PA analyses^ 

o The uneven representation of departments across the 19 

fields Is evident In Table 2« Biosciences were represented 
by 22 departments^ for examplet but anthropology by only 3# 

Table 3 shows the distributions of the departmental samples 
according to the degree goals of the students Involved « The 
majority of samples ^(80 of 138) Included both prospective master^s 
and prospective doctoral students, 41 Included master^s students 
only, and 15 Included doctoral students only^ It should be recalled 
that flrst«*year' programs and evaluation procedures were reported to 
be comparable for both prospective master^s and prospective doctoral 
students In samples Including both«* 

«» 

Coding for sex and ethnicity was optional and quite unevenly 
available* Several samples, primarily In the physical sciences. 
Included sdme ''foreign students for whom English Is not the native 
language^'' Only scattered data were available for women, minorities, 
and foreign students* 



Principal Study Findings 

The Cooperative Studies were concerned primarily with assessing 
the relationship of Individual predictors, as available, to first-year 
graduate grade point average (Graduate CPA)* As Indicated earlier, 
other performance measures were sometimes provlded«»«-f acuity ratings, 
end-of-year examinations,, grades In critical courses or course 
sequences, and the like***-but the general Graduate GPA was the 
"common criterion" in essentially all the studies*** 

The correlation corefflcient was employed as the Index of 
relationship between a predictor and the GPA criterion* Called a 
validity coefficient when used to express the relationship between 
standing on an admissions or predictor variable and standing on a 
performance or criterion variable, the correlation coefficient is a 
familiar index that ranges in value from *00 (indicating no relation-- 
ship at all between two variables) to + KOO (indicating either 
a perfect positive or a perfect negative association)* In studies 



* Findings for different types of programs arc described in a later 
section (cf«, section on subgroup validity)* 

** For results of one study Involving a ratings criterion in addition to 
Graduate GPA, see Appendix C-1* 
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Table 3 



DlHtributlun of Sampies According to Degree 
Goals of Students Involved 



Field/ 
Uc>(>artncnt 




of t}ani>les with: 


Master *s 
only 


naSLer 8 a 

doctoral* 


only 


Biosciences^ 


7 


14 


•••• 


ViiteiDinLry 


1 


10 








.5 




• 1 1 # { ^C 

rUi V neinav ics 


I 


4 




Physics 


1 

X 


2 


2 


(»eoii)f»,yj i»copnyr»xc8 




2 


2 






5 


V 


Anthropology 




3 




Education^ 


5 


2 




hnglish 




5 


1 


History 


3 


5 


2 


Political Scicnoes^* 


1 


3 




Psychology 


A 


3 


5 


Sociology* 


4 


3 




Library Sciences 


2 


1 




Kine ArLs*» 


4 


2 




MUHiC 


•1 


2 




Phi li>Kophy 




4 


1 






5 




Iiaal 


41 


HO 





*Witn<uit s1>;nitiiant exi-tptu>n» first-year progiam;» and evaluation pro- 
itam.'r; werr ropi>rti'd to hv lomparabU' in theso samples which included 
iH>ih prospiulivo m^stt'r*^ and proupectivc doctoral candidateSt 

•Mmhuit s iKtMn*>>;raphv, M.irint- Knvlrontnt nt.tl Srit^mc, Allitd UtMlth Srlencc 

*Mn. luiU-s l.n^\inovt ln>». and Fai ilitios Xana>ui-ent 

Mn. luth v (.ouputn Si ieni't\ AppHid M.tt*» & Statisiiis 

**hKlndi:» V.>.ation.il and Adult l.dm at imi, tdu*alii»nal Ainiinist rat ion 

**hu huU-H Puhlii Admini-.trat ion 

hn. luai?» Sov ial Woik, l-rh.ui Planniu>-., PuMio Poli^v Studiofi 

»*Im luai :» ;»pv»i h ana Tht ator. luatua and ( i.mn.unii at ii>n. Spi't ih and Conummii at u-n. 
^ .ttuit na t isni 

*Mn. iudi-^. two tlf.pam^, .»n» ta-it:..nu»» -n*- Htibh, .*:id *Mif ;indi t t t-t t nt lat 
K»M»i>-.n I. in>*.n i; I »itul l.il»^att^l^ 
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involving standard admissions measures (such as tests of verbal or 
quantitative reasoning and an index of past academic performance) 
and academic criteria (such «. first-year grade point average or 
faculty ratings), obtained validity coefficients are expected to be 
(and lULnosf. always are) positive** 

» 

Collectivelyt correlational analyses In more than 130 departmental 
samples yielded a total of over 400 vaJLidity coefficients* ^Results 
of the departmental analyses, grouped by field are summarized in 
detail in Appendix Cr2«** For each departmental sa]iq>le> Appendix 
C-2 provides information regarding (a) the obtained validity coefficient 
for GRE«*Verbal> GRE«^uantitative> GRE-Advanced » and/or Undergraduate' 
GPA» as avaiiablet with respect to a, first-year Gl^A criterion^ 
(b) means and standard deviations of scores on predictor and criterion 
variablest and (c) the number of cases used to compute each validity 
coefficient* 

V 

An illustrative summary of findings for departments from one 
cluster of fields, (history > ^rea studies > and anthropology) is 
provided in Table 4# Certain characteristics of the samples and 
data alluded to above (eege> small N^s and missing data patterns) 
are clearly evident in the overall patterns of departmeiital findings* ^ 
More importantlyt however^ the results in Table 4 point trends 
that were common to each of the fields or groups of fields considered: 

o Fitstt the validity coef f i*cients for GRE and UGPA predictors 
were overwhelmingly 'positive ^ indicating that individuals 
with higher scores on GRE and UGPA predictors tended to 
have higher first<*year grades; 

o Second* this pattern lield fotf samples differing rather / 
markedly in level of GRE i^titude* Nbte* for example* that 
GRE^^Verbal validity coefficients Were positive In each of 
13 samples with mean GRE-*Verbal scores that spanned a 200 
point range-f^from a mean of 502 (School 204) to a mean of 
. 698 (School 035) e 



* In a recent summary of the results of over 600 validity studies 
Involving the Law Schaol Admissions Test and Undergraduate CPA as 
predictors and First Year Average in law school as a criteriont 
Schrader (1977) lists over 1,200 validity coefficients of which only 
11 were negative^ For additional discussion of factors involved in 
evaluating observed validity coefficients in small departmental 
samples see Appendix C-1, especially ppe l-8e See also Willinghara 

(1974)e 

** Appendix C-2 is a summary report, (Tabular Summary of Selected 
Validity Study Findings, March, 1978) which was prepared for partici- 
pants in the Cooperative Studies Projecte 
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Summary of Validity Coefficients for GRE and UGPA 
Predictors versus First-Year Graduate Grades: 
History, Area Stuiles, Anthropology 



School (coded) Pr<>dlctor mean & s.d. Validity <^°c"^ctent J*^*\^^-V . 

(N/ coefficient) gRE-V GRE-Q GRE-Adv UGPA GRE-V GRE-Q GRE-Adv UPGA mean & s.d. 

.39 ~ 3.56' 

0.55 

.20 .05 — 3.81 

0.24 

.14' — .02 3.51 

0.35 

.26 -.10 .40 3.47 

O.Sl 

.45 .15 .63 3.64 

0.31 

.36 .72 .38 3.47 

0.51 

Hlscory 

3.55 .20 .20 - .20 3.42 





School 035 
(36,36,17,—) 


698 
78 


603 
98 


595 
54 




- History 
,27 

Hl«tory 


School 046 
(31,31,13,—) 


615 
87 


535 
98 


563 
60 




,10 
History 


School 080 
(36,36,— ,38) 


601 
113 


541 
107 




3,39 
0,42 


,16 • 
History 


School 097 
(30,30,26,32) 


612 
87 


538 
94 


563 
76 


3e56 

0.34 


,06 
History 


School 103 
(34,34,26,26) 


594 
91 


552 
121 


557 
55 


3e42 

0,38 


,38 
History 


School 123 ■ 
(27,27,14,27) 


589 
87 


520 
111 


534 
81 


3,14 
0,51 


.58 



School 145 647 546 

(48,48,-, 118) 99 126 0.35 



0.43 

Asian Studies 

School 145 651 574 - -66 . 35 ~ .12 3. 33 

(37,37,~,55) 

History 

school 221 649 602 594 ..o. .29 .22 .36 .53 r.K 

(43,43,43.32) 80 107 73 0.28 ^^^^^^ 

school 231 603 509 .7U .^2 .45 .02 .43 3.46 

(26.26,22,26) 

Anthropology 

School 009 642 582 07 .11 - - J.j: 



(37,37,—,—) 



651 


574 




3,32 


78 


103 




0,52 


649 


602 


594 


3,62 


80 


107 


73 


0,28 


603 


509 


570 


3,34 


109 


121 


79 


0,50 


642 


582 






90 


85 






649 


580 




3,62 


118 


109 




0,30 


502 


483 






14^ 


113 







Anthropology 
.41 .30 — .06 3.58 



school 145 649 580 - ^.o^ 

(39. 39, --,47) 118 109 0.30 

f-^ij Anthropology 

School 20.i 483 31 .20 - - ^.-5 

(1-^.H. 
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Thir4f the mean Graduate 6PA (on a scale In which A « 49 ^ 
B » 3t # « #t F « 0) was higher than 3*4t or approximately 
, ••B+t" In 11 of the 13 samplest reflect^lng a cchtislatentK 
tendency for grades to be restricted largrly to A's and 
B'ft— a pattern that trends toUlmit the overall ••p>^«<iict- 
ability'* of differences In grades^ 

The general tendencies gand trends noted above for departmental 
samples from history » area studies » and anthropology are discernible 
in study findings for departmental samples from other fields (see 
Appendix Tables l-ll)* These findings, generally summarlzedt 

indicate that: ' 

o First-^time graduate students with higher scores on GRE 

tests (of developed verbal and quantitative reasoning and/ . 
or achievement) or with higher undergraduate grad^ point 
averageSf tended to be better performers* on the average* 
than their lower^-scorlng counterparty* when performance was 
^ measured by grades earned during tt#^. first year of graduate 
study* This finding held for studies involving a vai4.ety 
of departments from a wide range of disciplines* 

PatteA& of Validity Coefficients by Field 

(• Results of the departmental analyses clearly support the 
fundamental premise underlying the use of GRE scores and measures of 
past academic performance (e#g** an undergraduate average) In 
'assessing the academic jiuallf icatlons ofr candidates for admission* 
namely* that these preadmissions measures should be positively 
associated with relevant measures of performance in graduate study 
(such as the first-year graduate grade point average)* \^ 

However* validity coefficients In small samples (such as those 
shown in Table A) have large sampling error* Moreover* In small 
samples one or two atypical data sets (called "outliers") can have a 
dramatic Influence on both the magnitude and the sign of an obtained 
coefficient** Accordingly* validity coefficients obtained in a 
given departmental sample (a) may not provide reliable information 
regarding the "true" degree of association between a given predictor 
and a given criterion— i*e* * coefficients will tend to vary substan- - 
Jtially from sample to sample in the same department* and (b) do not 
permit inferences as to the relative validity of two or more predictors* 

By aggregating or pooling data for several different departments 
in a given field (such as hiJitory)* however, it is possible to arrive * 
at more reliable estima^tes of validity coefficients for predictors 



* See study findings for School A* Appendix C-1* for evidence on 
this point; see also Appendix !)• 
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and examine at least tentatively variations in patterns of validity 
coefficients across different fields of study. One approach 1 to esti- 
mating validity coefficients in pooled samplies from several departments 
. * / in the same field involves the use of predictor and criterion variables 
that have first been departmentallv stand^lrdlzed , as decribed below. 



Validity coefficients based on standardized va riables. Pooling 
data on GRE scores,. UGPA, and Graduate GPA for several departments 
is complicated by the fact (a) that departments differ in levels 
of scores on the predictor variables, and (b) that the criterion 
variable (Graduate GPA) does not have a standard metric from., 
one department to the next. Graduate GPA scales tend to reflect 
primarily departmental "norms" or standards; accordingly, it does 
not follow that differences in the average level of grades awarded 
across department J reflect "real" differences in level of student 
academic output. "A department, with "lower" mean GRE scores, for 
example, may have "lenient" grading standards, and a department with 
a "higher" GRE mean may have, "strict" standards (as suggested, for 
example, by data for the history departments in Table 4). In any 
event, it is not possible tc generate interpretable validity coeffi- 
cienta simply by combining the original predictor and criterion 
data for several departments. ^ 

However, the problems posed by differ^ces in grading scales 
and in levels of scores on GRE and other ^edictors may be dealt 
with by converting all predictor and crH?ferion variables to a 
standardized scale within each department , prior to pooling. That 
Is, the GRE scores, UGPA, and Graduate GPA scores for individuals 
can be expressed as deviations from departmental means in standard 
deviation units . After standardization, each variable would have a 
mean of zero and a standard deviation of unity within each department; 
these standardized scores would be comparable in meaning for Ifidlviduals 
without regard to department. Following the departmental standardization 
of all variables, Interpretable correlation coefficients could then 
be computed based on the standardized variables using data for all 
individuals from all samples . 

» 

The coefficients In Table 5 -represent, for each of 19 fields or 
clusters of fields, predictor-criterion correlation coefficients 
based on departmentally standardized variables, using data for all 
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departmental samplea combined** The coefficients may be Interpreted 
directly as reflecting the degree of covariation between (a) standing 
on a designated predictor relative to departmental norms and (b) standing 
on the Graduate GPA criterion relative to departmental norms ♦ Also shown 
In Table 5 la the nuikiber of cases upon %ihlch each coefficient Is based# 
For example, tAe coefficient of #19 for GRE«»V vs* GPA (departmentally 
standardized) for Biosciences la based on a total of 580 cases from 22 
biosciences departments (see Table 2 for the number of samples pooled)* 

Certain trends are noteworthy. Including the following: 

o The fields In Table 5 may be thought of as tending to 
make either primarily quantitative or primarily verbal 
demands on students* For example, demands on students In 
biosciences, chemistry, engineering, mathematics, physics, 
geology, geophysics, and economics may be thought of as^ 
more quantitative than verbal, whereas In the remaining 
fields demands may be thought of as more verbal than 
quantitative* Inspection of the validity coefficients In 
the table reveals a tendency for GRE<-Q to have higher 
validity than GRE^V In the quantitative fields (except 
mathematics), and for GRE^V to have higher validity than 
GRE-Q In the verbal fields (except psychology and library 
science, both of i^lch have some quantitative emphases)* 

In evaluating the coefficients In Table 5, It should be recognized 
that they are conservative estimates of the degree of covariation between 
the predictors and ^level of academic output" across the entire range of 
talent represented In the pooled samples~l*e* , these pooled coefficients 
based on standardized variables are lower than those that would be 
obtained if all the individuals Involved were competing In the same 
department * 



* One way of illustrating the ''meaning" of correlation coefficients 
of differing magnitude is to show for selected validity coefficients 
how the relative standing of individuals, on a predictor tends to 
vary with their relative standing on the criterion under consideration, 
as in the exhibit below (adapted from Schrader, 1971, Table 5*5): 

Standing of Expected standing of students on criterion 

students variable (in percent) when: 

on a 



predictor 


r 


« .20 


r 


» .30 


r 


» .40 


variable 
















Low 


Mid Top 


Low 


Mid Top 


Low 


Mid Top 




20% 


60% 20% 


20% 


60% 20% 


20% 


60% 20% 


Top 20% 


13 


59 28 


10 


57 33 


7 


55 38 


Mid 60% 


20 


60 ''20 


19 


62 19 


18 


64 J.S 


Low 20% 


28 


59 13 


33 


57 10 


38 


55 7. 
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Table S 



Validity Coefflclants Estlmted in Pooled Departmental Samples 
Using Departmentally Standardised Variables 



Field/ 

Departments 
poo lea 




* 

Size of pooled sample 




Validity Coefficients 




6RK- 
Aptitude ' 


GRE- 
Advanced 


UGPA 


GRE-V 


GRE-q 


GRE- 
Advanced 


UGPA 


Biosciences A 




580 


' ■ ' m 
219 


419 


.19 


*25 


• J/ 


.24 


Chemistry 




389 


219 


370 


.09 


«31 


• 39 


.31 


Engineering^ 




202 


43 


91 


.28 


♦ JU 




.20 


Mathetiiiitics<^ 




15A 


34 


32 


.32 






.30 


Physics 




183 


176 


151 


.05 


♦ 16 


.19 


.29 


Geology; Geophysics 




98 


75 


39 


.05 


*06 


.11 


37* 


Economics 




204 


110 


125 


.09 


♦ 34 


.45 


.27 


Anthropology 




95 




47 


.26 


♦ 21 




.06* 


Education<^ 




292 


59 


332 


.18 


♦ 12 


.54 


.24 


English 




190 


122 


144 


.41 


♦ 24 


.48 


.22 


History 




348 


160 


284 


.31 


«26 


.21 


,30 


Political Sciercese 




181 


75 


157 


.43 


♦ 34 


.49 


.18 


Psychology 




521 


279 


306 


.24 


♦ 26 


.37 


.22 


Sociology^ 




287 


43 


146 


.43 


♦ 30 


.54 


.55 


Library Sciences 




117 




118 


.32 


♦ 52 




.33 


Fine Arts* 




2AA 




219 


.33 


♦ 26 




.31 


Music 




178 


85 


33 


.24 


♦ U 


.21 


.23* 


Philosophy 




82 


17 


40 


.25 


♦ 04 


.23 


.56 


Languages^ 




€8 


31 


28 


.31 


♦ 20 


.45 


.28 



Total 



4433 



1749 



3081 



• Not Computed- 



NOTE: Validity coefficients are based on departmentally standardized variables. Tlie tot.il numbor of 
cases per coefficient is shown under pooled sample size^ See Table 2 for the number of departmental 
Siimples for which data were pooled* 

♦Coefficient based on one sample only^ 

^Includes Oceanography^ Marine Environmental Sclencis Allied Health Science 

^Includes Engineering and Facilities Management 

^Includes Computer Science, Applied Math and Statistics 

<^ Includes Vocational and Adult Edu.-tation, Educational Admini jurat ton 

^Includes Public Administration 

^Includes Social Work, Urban Planning, PubHc PoUov Studli^s 

^Includes Speech am* Theater, Drama and Communioat Itnu SpcMv^h and Communication ♦ .Jnurnali«m 

^Includes two Hispanic* one Oermanio, one French, .md oae undifferentiated Forel>in Languages and Literature 
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o In 15 of the 19 fields^ one or more 6RE predictors ^tended 
to have somewhat higher validity than UGPA* 

o The overall pattern of coefficients suggests the potential 
Importance of 6RE Adval^ed Tests as predictors of flr^t-year 
perfbrmance in graduate^ study* Advanced Test scores » the 
least frequently reported predictort tended to yield 
validity coefficients somewhat larger than those for 
GRB Altitude or UGPA* 

Additional evidence bearing pn tHe relative Importance of ' 
Individual predictors Is provided In Table 6^ which shows the 
distribution of sample validity coefficients (based on 10 or more 
cases) for the respective predictors In departments from 'Verbal'' 
and "quantitative" fields as defined above* 

o The pattern of validities for quantitative fields 
suggests a '^primary role for Advanced Tests » followed 
by Quantitative Aptitude and Undergraduate GPAf while 
for verbal fields the Verbal Aptitude score supplants 
the Quantitative score In this pattern* 

As will be seen later» the ORE predictors and the Undergraduate 
CPA each tend to provide some unique Information about performance 
potential* 

in evaluating the magnitudes of the coefficients in Tables 5 
and 6t It Is Important to recall that the Graduate GPA criterion 
frequently was severely restricted In rangCf being weighted almost 
always In the direction of "higher grades" — usually only B's and 
A'^s* In the circumstances » It Is reasonable to Infer that observed 
validity coefficients are somewhat lower than would be the case If 
differences in student performance were more rigorously and reliably 
assessed by routine grading procedures* 



Comparison with Other Validity Study Findings 

Tables 3 and 6 provide summary data Indicating the typical 
levels and patterns, as well as the range, of validity coefficients 
for ORE and UGPA predictors In samples from a variety of fields or 
groups of fields* Table 7 relates findings of the Cooperative 
Studies for selected fields, based on cohorts entering In 1974 and 
1975, to findings of studies conducted during the period 1952-72, as 
summarized by Wllllngham (1974)* A general similarity In the overall 
patterning of median validity coefficients for the respective predictors 
is evident for studies that were conducted during two different periods 
and that involved different samples* The validity coefflciencs for GRE 
Advanced Tests, which reflect aptitude, motivation, and substantive 
achievement In specific fields of study, are typically, somewhat higher 
than those for other predictors in both periods^ This fact provides 
additional evidence of the potential importance of these tests* 



T«tble 6 



Distribution of Validity Coefficients for Groups 
Qf "Verbal" and "Quantitative" Fields 







Verbal 


fields* 






Ouantltative fields** 




Level of 
validity 


GRK-V 




ORE- 
Advanced 


UGPA 


aR>>v 


. GRE-<i 


GRE-^ 
Advanced 


UGPA 


.bO + 


2 


2 


A 


3 


1 


1 


1 


1 


. 30 • ,59 


8 


A 


5 


A 


1 


6 


6 


2 


.40 • ,49 


16 


8 


5 


5 


A 


11 


A 


5 


. 30 - ,39 


10 


11 


A 


10 


8 


15 


A 


10 


.20 • ,29 


17 


18 


A 


11 


16 


17 


5 


9 


.10 - ,19 


8 


15 


5 


8 


10 


8 


5 


4 


.00 - ,09 


A 


A 


3 


A 


11 


10 


2 


2 


N««>:«tlve 


5 


8 


1 


1 


11 


5 


1 


5 


Total 


70 


70 


31 


Ah 


63 


63 


29 


38 


Median Coefficient 


.31 


.25 


.35 


. 30 


.20 


.31 


.3A 


.29 



NOTE: IncludfH data tor san^)les of ten or nk^ro casfs. 

*Biosc fences* cheinistrv» enginoerlngt nuithonuit icH» physics* Keolojw, economics and other fiolds (cf.» notes 
to T.ihlf 5). 

**rnrHrth, hfHti»rv» s«>r iolovAS x'.ovcrntn*»nt and p*>litical scitMict^s, psvcholo>:y » fdm'.it i4>n» l,in>-uar*-s» anthropoloKy 
and other fields (if.» n.»tos to Tablf S). 
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Table 7 

MedUn Validities Obtained in Cooperative Validity Studies 
Sanplea in Selected Fielda vith MedUn Validities Ob- 
tained in Earlier Validity Studies* 



Field (s) 



Period covered 
by 
studies 



Median validity (nunber of samples) 



GRE*-VerbaI 



GRE-^anti- 
Utive 



GRS Advanced UGPA 



Biosciences 

Chemistry 

Engineering 

Katheiaatics 

Psychology 

Education 

English 



"Verbal'' fields 
Social science 

••Quant/* fielda 



1974-1976 


.19 


(22) 


.24 


(22) 


.37 


(JO) 


.31 


(14) 


1952-1972 


.18 


( 7) 


.27 


( 8) 


.26 


( 5) 


.13 


( 2) 


1974-1976 . 


.19 


(11) 


.37 


(11) 


.41 


( 3) 


.33 


( 7) 


1952-1972 


.22 


(14) 


.28 


(13) 


.39 


( 9) 


.27 


( 7) 


1974-1976 


.26 


( 9) 


.38 


( 9) 






.14 


( 4) 


1952-1972 


.29 


(11) 


.31 


(10) 


.44 


( 7) 


.18 


( 4) 


1974-1976 


.30 


( 6) 


.29 


( 6) 


.40 


( 2) 






1952-1972 


.30 


( 6) 


.27 


( 6) 


.44 


( 5) 


.19 


( 4) 


1974-1976 


.18 


(13) 


.19 


(13) 


.32 


( 8) 


.20 


( 8) 


1952-1972 


.19 


(23) 


.23 


(22) 


.24 


(17) 


.16 


(15) 


1974-1976 


.16 


( 7) 


.20 


( 7) 


.53 


( 2) 


.30 


( 4) 


1952-1972 


.36 


(15) 


.28 


(14) 


.24 


( 6) 


.30 


( 5) 


1974-1976 


.30 


( 7) 


.18 


( 7) 


.40 ■( 6) 


.27 


( 4) 


1952-1972 


.21 


( 6) 


.06 


( 6) 


.43 


( 3) 


.22 


( 4) 


1974-1976** 


.31 


(70) 


.25 


(70) 


.35 


(31) 


.30 


(46) 


1952-1972 


.32 


(11) 


.32 


(10) 


.46 


( 5) 


.37 


( 5) 


1974-1976** 


.20 


(63) 


.31 


(63) 


.34 


(29) 


.29 


(38) 



Source of data for earlier studies is a sunnnary by Willlngham (1974) of studies 
during the period 1952-1972. Medians for 197A-1976 are from the Cooperative 
Studies* using a Graduate GPA criterion. Earlier validities are primarily 
from studies using Graduate GPA but other criteria were involved in some 
ca«tes* Number of samples on which medians are based 1h shown In parentheses* 



Cf*» Table ^ and related discussion* 
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ComparlBon of Validities at Graduate and Undergraduate LevelB ' 

Evidence regarding typical levels of validity coefficients 
obtained in the Cooperative Studies may usefully be compared with 
evidence from underRraduate validity studies that have employed a 
GRE--comparable measure (namely^ the College Board^s Scholastic 
Aptitude Test > which yields a verbal score and a mathematical 
reasoning score) and a measure of previous academic performance 
(high school CPA or rank in class) versus a first-year grade point 
average criterion* Such a comparison is provided in Table 8# 
Several points are important: 

o Results of the college«-f reshman levet studies are 

summarized so as to indicate how validity coefficients 
tend to be lower in samples that arie highly selected on 
verbal ability than in samples that are more represent- 
ative with respect to verbal ability* 

o Median validities for GRE-Verbal in primarily verbal 

fields and for GRE-Quantitative in primarily quantitative 
fields in samples of first-time graduate students (a) 
are equal to or higher than median validities for compar- 
able undergraduate predictors in samples of college 
freshmen that are relatively homogeneous with respect to 
verbal aptitude^ and (b) are not markedly lower than 
validities obtained in more representative college freshman 
contexts (e*g*» colleges usltig the College Board SAT)* 

o The most notic^&ble difference between undergraduate 

and graduate-level findings Is with respect to the validity 
of the record of previous academic performance: median 
validity for Undergraduate CPA is rather markedly lower 
than the median validities for High School CPA or Rank* 
Graduate students generally may tend to be relatively more 
highly selected on academic drive and motivation (which 
undergraduate grades reflect in considerable measure) than 
are college freshmen generally* 

In any events the findings in Table 8 suggest that despite the 
recognized limitations of first-year graduate grades (narrow rdnge, 
over-representation of •*higher marks, etc*), when they are employed 
as a measure of performance, validities obtained for ORE Aptitude 
tests are' similar to those obtained for comparable tests versus 
college freshman CPA in many undergraduate samples, especially those 
that are relatively homogeneous with respect to verbal ability* The 
validity of UGPA for predicting first-year graduate grades appears 
to be considerably lower than that of high school GPA or rank for 
predicting first-year undergraduate grades* 
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Median Validity Coefficients and RanRe of Coefficients tn Studies of 
Comparable Predictors and First-Year Grade Average 
Criteria in Graduate and Undergraduate Settings 



Predictors as 
appropriate 
to level of 
study* 



CRE studies 
in graduate 
school 
settings 

Median 



College Board SAT validity studies in 
undergraduate samj^les which were : 



High & homogeneous 
on SAT-V** 



Homogeneous Representative 
on SAT-V*** SAT-V scores 



Median (range)# Median (range) Median (range) 



GRE-V (verbal fields) 

SAT«*Verbal (undergraduate) 



.31 (yO)**'* 



^ ,22 (.11 to .AA) .31 (.15 to .A6) .39 (.26 to .5A) 



GRE-Q (quantitative fields) 
SAT-Math (undergraduate) 



.31 (70) 



— > ,2A (-,01 to .A6) .27 (,11 to .AO) .33 (.20 to -AS) 



GRE-Advanc ed ( gr ad ua t o ) 
(appropriate to field) 



.3A (70) 



Undergraduate CPA 
High School Record 



.29 (84) 



V ,40 (.^2 to .57) .AA (,26 to ,59) .53 (.33 to ,67) 



NOTE: GRE validity data are from the current Oooporativo Studies. Undt^rgraduate validity data an* t rora Schtador (1971). 

* The coeff icit»nts in coltiran 1 of the table refleot validity of GRf>Verbal In verbal lioldn^ f^RK-Quant itatlvo in 
quantltativif fi<*ldB» GRK- Advanced TcKtj; .ik appropriate to a field, and llndt»rrraiidate CPA with»>iit r«M»ard to field. 
The reimilning coefficients an* for the Scholi«tlf Aptittide Test (Verbal and MathenwitftMl) and the high sehnol 
record (either GPA or rank- In-class) In college freshnum samplert. \ 

♦^Studies in 18 undergradu*ite samples havlnv» an SAT-Verbal tnean above hOO and a standard deviati^m of h*y or h's« 
(college freshnuin level), 

♦♦^Studies In 93 samples of unuergraduate fr-shman men and women havin« SAT-Verhal 5;tandard deviations of lfs<t 
than 75. Median values repi>rt*'d separately for**tn«'n .md Wi>nt»n hv Srhrad«*r have het»n averav'ed for pr<'S»»nt at ion 
in this table, 

# Approxlnately 80 percent of the obtained ooef f Ici^nts vtvw within th«» ran/e •;jM't i t' i«d. 
I^Number of coeftic tents (samples) upon whieh each mi»iiian i*i ba*^vd« 

i 
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Validity fftf Subgroups 

In the samples submitted for the ORE Cooperative Validity 
Studies Project, it was not feasible to address systemetically 
and rigorously questions regarding the comparative validity of 
GRE tests for important subgroups such as minority students, 
women, foreign students, or students classified according to type 
of degree program (e.g.. terminal master's, master s-Ph.D. sequence, 
or doctorate only). The inability to address these questions was due 
primarily to small numbers or the failure of departments to identify 
.subgroup membership. However, some scattered analyses with very 
small numbers of cases were carried out. Only limited conclusions 
can be drawn from these findings, which are summarized generally in 
this section. 

Minority studen ts. Some useful albeit limited evidence 
bearing on the validity of GRE scores and undergraduate grades for 
undifferentiated "minority" students (i.e.. considering Black. 
Chlcano. Puerto Rican. etc. together) is provided by findings in 
several very small samples from six graduate schools, summarized in 
Table 9. In samples from 15 different departments, the number of 
minority students with predictor (ORE AptitUde) and "i^erion 
(Graduate GtA) data ranged fron: three to 20 students per department. 
Needless to say. given the small Ns involved, it is important to 
look primarily for trends, or consistencies that are discernible in 
the data. It is evident, for example, that validity coefficients 
tend to be positive in minority, nonminority. and pooled samples. 

In samples for School 221. separate analysis was not made 
of data for nonminority students only. However, in samples from 
Schools 097. 231. 132, and U5 data were analyzed separately 
for minority, nonminority. and pooled minor ity-nonminority samples. 
In almost every case, it may be seen that validity coefficients for 
GRE predictors were larger in the pooled sample than In the sample 
of nonminority students only. 

This important finding reflects the fact that minority 
students typically had substantially lower GRE scores an^ 
tended to earn lower grades during the first year of study than 
their nonmln«»rtty classmates. 

o Figures 1 and 2 provide graphic evidence of this 
phenomenon. Those figures show plots of GRE-Verbal 
scores versus Graduate GPA for minority students and 
small, r.m.l.wn samples of nonminority students in 
journalism (iichool 231) and psychology (School 097). 
respectively. Note that the points or other symbols 
representing predictor-crltorlon scores for minority 
students tend to be clustered in the lower left qucidrant 
of each figure, indicating "below average" GRE-Verbal 
scort'H and "below average" Graduate GPA. 
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Table 9 

CRE Aptitude Validity Coefficients In Sanples of Minority Students, Nonninority 
Students and Pooled Hiaority/Nomtinoriey Students 



School/ 
rieia or 


mnority 


> 

only 


nonmi no r i c y 


only 




Pool 6(1 




Depart iDent 


V 


Q 


V 


Q 




V 


Q 


School 097 
















Psychology ^ 
(a, 68, 76) 


19 

• 


02 


00 


-01 




14 


11 


^School 132 
















Psychology 
(10, 43, 53) 


77 


82 


23 


29 




52 


54 


School 145 
















^/ Psychology 
(20, 69, 89) 


jj 


33 


..19 


17 




35 


35 


Education 
(10, 40, 50) 


26 




27 


09 




33 


22 


School 231 
















Journal Isn 
(07,25,40) 




•11 




2o 




21 

•J 


34 


Scnool 2U4 
















Applied Math 


28 


52 


27 


07 




27 


27 


Spanish 


61 


00 


— 


— • 




37 


•14 


Music 


•83 


08 


17 


•08 




1* 


-04 


School 221 #1 
















Chemistry 
(03, 52) 


-34 


99 








-21 


42 


Psychology 
(09, 45) 


•27 


•38 










45 


History 
(04, — , 46) 


72 


12 








29 




English 
,(04» 55) 


67 


87 








44 


34 


Library Scl 
(04, — , 40) 


81 


72 








47 


59 


Hispanic Lang 
(06 » 14) 


51 


86 








55 


70 


Public Adtnin 
(14/—, 41) 


47 


17 








32 


54 



Note; Numbers in body of table, oppo«:Ite field designations, are correlation cooffUiffnts 
with docimiil omitted, Tlie criterion is <;raduate (JPA, 

Numbers in parentheses are Ns used to ronpute the coefficients. For example, the 
minoritv inalvses involved H cases, the nonmlnurity analyses 68 cases, and the pooled 
an.ilvses 76 cases in school 09 7 • 

Includes ei^ht foreign student s« 
P Includes four foreign students, 
## Nonminority data were not analyzed separately. 



m 



SCHOOL 231 



900 



700. 



600. 



SOO 



400 



\00 



Validity coefficient 
for 

Minority r « -.36 

Nonminority r - .08 

Pooled r • .21 



NONMINORITY I 
Mean V - 585>> ' 
CPA Mean « 3 . " 





A •! 



Not£: Each A represents a 
mTnorlty student with a given 
GRE-Verbal and Graduate CPA; 
cachB represents a nonmlnor- 
^ity student (small random 
sample ♦* only) ♦ 

Five minority students 
with no score had a mean 
Graduate GPA of 3*22« 



4 • 

O Mean V 
\il?A M^an 
MINORITY 
I 

I 



399 
3.29 



1^ 



2.0 
C 



2.5 



3.0 
B 

Graduate (!PA 



3.5 



4.0 
A 



Figure 1. Plot of GRE-Vcrbal scores vs. Graduate GPA for 
minority and nonminority students in Journalism 



So 



900 



750 



700 



650 



600 



550 



500 



45» 



200 



SCHOOL 097 



# # 




♦ • Nonminority 

Black 
O • Other minority 

Validity coefficient 
(GRB-Verbal vs. GPA) 

Minority r • .19 
Nonminotity r • .00 
Pooled r » .14 

' Mean values GRE-V GPA 




Minority 
Nonminority 
Pooled 



549 4.60 
629 4.83 
621 4.81 



It 
X 



1»0 



3.5 



4.0 

Graduate GPA 



4.5 



5.0 



Figure 2. Joint distribution of GRE-Verbal scores and Graduate 
GPA for minority students and a sample of nonminority students 
in Psychology 
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This evidence (a) that the GRE validity coefficients tend 
to be positive for minority students and (b) that minority students 
with GRE scores well below the aveK^ge for a given department tend 
to perform at a level that is also below average for the depart- 
ment, during the first year of study* Is consistent %fith evidence < 
from comparative validity studies in undergraduate^ and law school 
settings (see, for example. Linn 1973, 1975; Wilson, •1978)* 

In evaluating these findings It Is Important to recognize not 
only that the findings are biased on very small samples In only a 
few departments » but also that the studies employed only first -year 
performance criteria^ This latter consideration Is esfeclally 
Important f torn the point of view of assessing ^he validity of 
preadmissions tests for minority students for vhom^ it may be argued » 
first-year performance may not provide a clear Indication of 
performance potential as a result of special problems of transition 
from undergraduate to graduate study** Studies employing criteria 
reflecting performance beyond the fir8t**year in graduate school » as 
* well as additional first-year studies^ are needed in order to 
provide more comprehensive evidence regarding the validity of GRE 
scores and undergraduate grades for minority students* 

Women * Coding for sex» as well as for minority status^ was 
optional* Although the. number of women identified was somewhat 
greater^ typically^ than the number of minority students^ sample 
size militated against routine analysis and reporting of data by 
sex* Limited analyses by sex indicate patterns of correlational 
validity for women that appear to be roughly similar to those for 
men* Data for several departments in each of two graduate schools 
are shown^ illustratively » in Tables 10 and II* Median validities 
for GRE-V> GRE-Q> and UGPA, across departments are summarized below^ 
by sex* ^ 

Median coefficient 
Men Women 



School No> samples GRE^V GRE-Q UGPA GRE-V GRE-Q UGPA 
221 ( 8) .37 .42 ,30 •25 ,53 ,40 

145 (17) .33 *33 ,33 *38 ,43 .31 

i 

These data, of course, do not permit conclusions regarding the 
relative validity of GRE and/or UGPA predictors for men and women, 
respectively. In the two schools Involved* However, the observed 
general trends are consistent with the expectation that GRE scores 



* Tlie validity of preadmissions tests for predicting the long- 
term performance of minority and nonmlnorlty students has been 
explored in recent studios at the undergraduate level (e*g»t 
Warren, 197b; Wilson, 1978) • These studies suggest that conclu- 
sions reached on the basis of comparative validity studies using 
the first year (IPA tend to hold for longer-term cumulative GPA» 
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Table 10 • . 

Correlation Predictors vith Gr«4M«te CPA 

by l>tp«rt»ent and by Sei^ * 
(School i*^> 

Oepartttent CR£>Varbel cafc^uentitativc Vftjf rUTft^y^f g^A 

r M Total F • M Tot#l F M Total 



Chemistry* 




• 45 


•41 


.39 


• 4$ 


• 46 


.36 


• 33 


• 36 


Biology* 


• 37 


• 44 


• 29 


• 91 


• 26 


.37 


.23 


,27 


• 23 


EconoAlct* 


.13 


.23 


.17 


•49 


• 33 


• 37 


.64 


-l04 


•02 


Philosophy* 


• 23 


.32 


• 29 


-.$6 


.06 


-•01 


•61 




• 77 


* 

Psychology 


• 49 


.25 


• 35 


.57 


• 29 


• 35 


.21 


• 13 


• 17 


Anthropology 


.16 


.57 


.41 


.13 


• 44 


• 30 


-.01 


• 11 


•06 


AttlAth &t.tullM 


• SV 


.70 


.66 


•58 


.35 


• 35 


• 10 


•05 


vl2 


History 


.62 


.09 


.20 


•4g 


• 10 


.20 


.31 




• 20 


English 


.u 


.35 


.23 


.34 


-•04 


• 18 


• 13 


.^.01 


.08 


Public Policy 




.30 


.50 


•91 


.48 


•69 


• 61 


.60 


• 54 


Socio loirv* 


• 30 


.40 


. 44 


•05 


•44 


• 26 


• 26 


.61 


•4 7 


Political Science* 


»44 


»48 


•44 


•99 


.43 


.\t 


-.40 


.34 


• 30 


Natural Resources* 


.71 


.23 


• 33 


•09 


.36 


• 31 


.74 


.14 


• 21 


Urban Planning 


• 36 


.26 


.27 


.43 • 


•03 


• 29 


.72 


.45 


.54 


Education 


• 36 


.17 


• 33 


• 19 


.32 


.J2 


• 25 


.35 


.26 


Libtary S<lrntr 


.45 


.55 


• 46 


• 60 


•65 


.59 


• 48 


.52 


• 46 


Speech 


*U 


.33 


. 3^ 


.11 


. 19 


• 13 


.56 


.02 


• 30 




.38 


.33 




.4^ ^ 


^ .33 


.31 


.31 


.33 


.23 



ar...fflr1rntrior "^^mrn ha«rd on\eii« than 10 canej. Ms for woiDen ranged ir^ 
tlve to m: <«T ttale». Na ranged fum 16 to 5l» inclusive. 




Table 11 



Corralation of PrVdictora with Craduate CPA 
by Dtpartmant and Sex 

(School 221) 



Predictor/ 
group 


Chetft^-'Psy- 
iatry chol- 
ogy 


Misr 

tory 


Eng- 
lish 


Lib- 
rary 

Scl 


Pranch Mis- 
fMmic 
Une 


Public 
Adain 


MedUn 
validity • 


CRK-V ( M ♦ F) 


-.21 


.42 


.27 


44 


• •» r 


.20 


.55 


.32 


(.43) 


Teaal« 


.27 


.14 


. «a 


. ^* 


46 


-.05 


.24 


.56 


(.25) 


Hale ' 


-.21 


.50 


.32 


.40 


.40 


•3> 


.92 


.26 


(.37) 


CRE-Q (M + F) 


.17 


.45 


.22 


:34 


.59 


.43 


.70 


.24 


(.38) 


Female 


.19 


.73 


.09 


• 32 


.62 


.46 


.65 


.61 


(.53) 


* Male 


.11 


.40 


.25 


.44 


.45 


.48 


.88 


.20 


(.42) 


Glt£*Adv <M ♦ F) 




.36 


.40 


n.a 


. .36 


.57 


.50 


(.37) 


Fcnale 


.37 


.38 


.50 


.44' 




.40 


.54 


.31 


(.39) 


Mftle 


.14 


.31 


.20 


.41 




.34 


.69 


.67 


(.32) 


UCPA ( Ik'*' F) 


<37 


.22 


.51 


.29 


.15 


.30 


.26 


.04 


(.27) 


Faaale 


.46 


.11 


.69 


.43 


.34 


.38 


.2^ 


.68 


(.40) 


Hale 


.41 


.31 


.43 


.21 


• 29 


.11 


.57 


-.07 


(.30) 


Kaxioitfli N 


52 


45 


46 


54 


40 


20 


14 


40 




HinUsuB N 


14 


14 


7 


17 


11 


7 


5 


4 





ii 
r 



Note: Advanced Test scores not available in Library Science. 
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and UGPA should have roughly coaparable predictive yalldlty for both 
men and wo^tenA More systeaatlc aasessnent la contingent upon nore 
. repreaentatlve data* 

Foreign Students * In several departnental sanples with a 
relatively high proportion ot ''foreign students for whom English Is 
not the native language" (as Identified by a department), data were 
analyzed separately for foreign and nonfor^lgn students despl|e the 
reduction In .sample slae entailed by such analysis becauijp^df the 
potentially depressing effect of lack of fluency In English on test 
performance* Such samples typically were froo* mathematics, science,- 
or engineering departments In which foreign students exhibited ' 
(a) "depressed" 6RE-Verbal scores, usualljr welllbelow the depart- 
mental average, but (b) .Quantitative* and/or Advanced Test scores 
' that tended <o be comparable, on the average, with those of their 
nonfOrelgn classmates* 

Results of several limited anal>^ ab for foreign and nonforelgn 
samples. In chemistry and engineering (see for example. Appendix C->2, 
Institutional Summary Report, Tables 2 and 3) Indicate that patterns 
of GRE.:Corre^atlonal validity were roughly comparable for foreign 
and nonforelgn students* The general pattern of findings suggests 
that GRE-Quantltatlve scores are comparable for foreign and nonforelgn 
students entering quantitative fields but that GRE-Verbal scores are 
not* The evidence provided by the current series of studies Is • 
consistent with and In a limited way extends evidence from studies 
of the performance of foreign students on the 6RE Aptitude Test 
(Harvey & Lannholm, 1961) and of the relationship of GRE Aptitude 
Test scores to first-year graduate grades In four graduate schools 
(Harvey & Pitcher, 1963), and In a sample from 24 graduate schools 
(Sharon, 1971)* 

In essence. It would appear that "depressed" GRE-Verbal scores 
of foreign students for whom English Is not the native language do 
not reflect accurately their performance potential relative to 
nonforelgn students In quantitative fields, although among foreign 
students differences In GRE-Verbal Aptitude tend to be positively 
associated with differences In graduate grades (e*g*, Sharon, 1971). 

Questions regarding the comparative performance of and the 

validity of GRE Aptitude and othe*- tests fot foreign and nonforelgn 

students In primarily verbal fie] do not appear to have been 
addressed systematically* 

Degree-level * Among the departmental samples involved in the 
basic one-year validity studies, 37 Included prospective master's 
candidates uuly, 76 dncluded both prospective^ master's and prospective 
doctoral candidates, and only 13 (six from one institution) Inclu'ted 
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prospectlv« doctoral candldatM only** 

o With isolated exceptions in departnental sanples that 
included both master's and doctoral students* first-yeac 
{irograms and . evaluation procedures were reported by the 
respective departnents to be comparable for both groups of 
candidates* In situations in which a given first-year 
criterion (such as Graduate GPA) reflects differences in 
performance based on a comparable set of tasks, kncurledge 
of the degree-orientation of the students involved is not 
essential to the orderly interpretation of validity coeffi- 
' cients* 

In the general summary of validity coefficients by department, 
the degree orientation of students in each sample as reported bv the 
department is indicated (see Appendix C-2, Summary Report, Table 
1-11). With the following departmental designations as a basis for 
classification, selected median validity coefficients were determined 
for "master's only," "master's and doctorate," and "doctorate only'' 
samples, as follows: 

a) median validity of GRE-Q versus GPA in 58 samples from 
primarily "quantitative" fields ( cf.. Table 6), and 

b) median validity of GRE-V versus GPA in 68 samples from 
primarily "verbal" fields (cf.. Tables 6). 

The results, shown in Table 12, indicate that predictor 
standard deviations and validities tended to be lower in doctoral 
samples than in either master's samples or samples that included 
both master's and doctoral cudents. 



* In explaining the very small nuinber of "doctorate only" samples, 
it is important to recognize that the study, by design, was restric- 
ted to first-time graduate students only. Many departments rely 
heavily on the recruitment of master's degree holders to obtain 
their prospective Ph.D. candidates. One departmental chairman, 
commenting on the small number of cases for which he could supply 
data, expressed surprise at "discovering" how few of his doctoral 
students had begun their graduate work in the department. The 
Cooperative Validity Studies Project was concerned with a clearly 
defined population that did not include students admitted to Ph.D. 
programs after having earned a master's degree or equivalent. It 
should not be assumed that work completed by such individuals during 
their first year in a program is comparable with that completed by 
first-time graduate students during their first year in a program. 
Validity study models for master's-holders entering doctoral level 
programs will be needed in order to deal with this general set of 
circumstances. 



It in important to keep in mind' in evaluating these result n that 
a very small number of doctorate«^nly samples is involved and that 
several of these were from only one graduate school; consequently^ 
specific detail should not be considered significant* Nonetheless» 
the observed Pattern is of interest > because it suggests that the 
lower median validities for the ^Moctorate only'* samples in this 
series of studies may be due primarily to greater restriction of 
range on the respective predictors in these samples* As indicated 
in Table ]3> median validity tends to decrease as predictor standard 
deviation v.dcreases without regard to degree^level of samples* More 
representative data will be needed to determine how general this 
pattern may' be* 

Summary: subgroup analyses # All findings with respect to 
^'subgroup validity'* should be viewed as suggestive only^ and as 
incidental to the primary objectives of the Cooperative Validity 
Studies Project* Uncae emphasis should not be placed on specific 
detail in evaluating the findings* Results for very small samples 
suggest that in graduate school » as in other academic settings t 
standarcl test scores are positively associated with grade point 
average for minority as well as for nonminority students and that 
lower-than-*average test scores for minority students presage lower-^ 
than«»average. first«»year grades* 

o In contrast^ limited analysis suggests that lower than 
average GRE-Verbal scores for foreign students (for whom 
English is not the native language) in heavily quantita- 
tive fields probably do not consistently presage lower 
than average first-year performance for foreign students 
relative to the departmental average* The first-year CPA 
level for foreign students appears to be roughly consistent 
with their average level oh GRE-Quantitative Aptitude* 

o Patterns of validity for women and ment respectively^ 
appear to be roughly comparable^ as expected* 

o And f inally> trends observed in connection with the 
analysis of validities for degree-level subgroups are 
consistent with famlllsir res tr let ion-of -range axioms 
(cf*> Appendix C-l> pp* 3-6; also Table 8 and related 
discussion) * 

However > In none of the subgroup analyses Involving woment 
ralnorlty» or foreign students could systematic attention be given to 
the many complex questions that are Involved in the rigorous determi- 
nation of the comparative validity and "fairness" of preadmissions 
measures for the respective subgroups (Llnn> 1973)* 

It is important to recognize that building a reliable body of 
empirical evidence bearing on subgroup validity will require the 
participation of graduate schools and departrnents In cooperative 
validity studies designed especially to collect data on the subgroups 
of Interest* 



/ 
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CRE Median Validities and Standard l>avlatii.. s for Sanplea Clasaified 
According to Degrea Oriantation of Students 



Verbal fields* Quantitative fields** 



Type of . No. of GRE-V CRE-V No^ of CRE-Q CRE-Q 

sasple dept. S.D. validity dept. . S«D« validity 

{wed ian) (med ian) (med Ian) (med ian) 

Master's only * 22 94 .?U 15 8S .25 

Master's 4 doctor's 38 97 .32 38 82 .34 

Doctor's only 8 80 .25 5 65 .00 



All samples 68 95 .31 58 82 .31 



*Biosciences» chemistry, engineering, mathematics, physics, geology and geophysics, etc. (cf.. 
Table 6 and related discussion)* 

**£nglish, history, psychology, languages, government, sociology, education and other fields 
(cf.. Table b and related discussion). 



Table 13 

Median GRE Validities for Samples Classified According 
to Size of GRE Standard Deviation 



Aptitude Quantitative fields V er bal f i elds 

Ktandard No. of GRK-Q No. of o'rE-V 

deviation GRE-Q validity GRE-V validity 

S . D. (median) S . D, (median) 



100 -f 12 .30 27 ,35 

60 - 99 20 .37 31 .30 

60 - 79 20 .25 12 ,20 

Below bO 6 ,15 0 

Total 58 .31 70 .31 



NOTE: Grouping of fields as for Table 12* 
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PART IV* THE PROBLEM OF COMBINING PREDICTORS 



IN SMALL GRADUATE DEPARTMENTS 



In validation research generally, it has been found that 
an appropriately weighted composite of scores on standard admis- 
sions tests and an index of past academic performance normally 
yields a higher validity coefficient (coefficient of multiple 
correlation) than either test scores or the past academic record 
considered separately* In large-sample studies, questions regarding 
the most appropriate weighting of admissions variables for the 
purpose of predicting a specified criterion are addressed directly 
by using appropriate multivariate methods, princ4.pally multiple 
regression analysis* 

Given a criterion such as first-year 6PA and scores on several 
predictors, the basic output of validity studies employing multiple 
regression analysis includes, in addition to validity coefficients 
for each of the predictors considered separately, 

1* a multiple correlation coefficient reflecting the relationship 
of all the admissions variables, considered Jointly, to the 
criterion; 

2* standardized regression weights (called beta weights) 
Indicating the contribution of each admissions variable 
xn an optimally weighted composite-predictor; and 

3* a regression equation specifying the (multiplier) %reight8 
to be applied to the scores on the admissions variables in 
order to obtain a composite-predictor score that is 
optimally-weighted for predicting the criterion variable 
under consideration* 

In large samples, the multiple regression model provides a 
systematic basis for determining how much each of several admissions 
variables contributes to the overall effectiveness of prediction, 
and the multiple regression equation summarizes scores on several 
admissions variables by giving to each score a weight that reflects 
its unique contribution to an optimally weighted composite predictor* 

In practice, regression weights are estimated in a given 
sample, such as one year's entering class, and applied in summarizing 
the admissions scores of candidates for admission to subsequent 
classes* Even in large*r samples, there are questions regarding the 



stability of the estimated regression Heights (e#g#t Vainer 1976/ 
1978)* However t especially in small samples the results of a regres^ 
sion analysis tend to reflect too closely possibly idiosyncratic 
patterns of interrelationships in the sample data (*^oyerf itting'* the 
data)» and thus do not provide reliable estimates of /the ''true'^ or 
population weights* Sampling error for observed regression weights 
in small samples is greats and weights developed in successive 
small samples of first--year students will tend to fluctuate widely* 

Because of the consistently small size of the departmental 
samples that are involved in graduate«*level validity studies* it is. 
not feasible to use multiple regression analysis routinely in order 
to determine '^optimal weights" for available predictors (e#g#» 
GRE-V» GRE-*Q» and Undergraduate 6PA or U6PA) that could be used by a 
department to form a locally relevant composite predictor* There is 
every reason to believet however^ that a combination of GRE and UGPA 
should lead to improved prediction of a given criterion* Uillingham 
(1974)» for examplet reported a median multiple correlation of #45 
for 24 departmental samples for a combination of 6RE-»Vt GRE«-Qt and 
UGPA» optimally weighted for predicting f irst«»year Graduate GPAt 
whereas median validities for these predictors considered separately 
were in the ^20 to #30 range # In the Cooperative Studies t multiple 
correlation coefficients were reported illustratively for some 
departmental analyses** In 27 samples from a variety of fieldst 
the median multiple correlation for the same set of predictors with 
respect to the CPA criterion was •43t as compared with median 
validities in the #30 to #35 range for the predictors considered 
separately* 

In view of the potential benefit (improvement in predictive 
validity) likely to accrue from combining predictors^ the problem of 
.^determining wieghts for GRE and UGPA variables that a given depart- 
ment might use to form a composite predictor is an important one* 
In small departmental samples (and other situations in which there . 
are insufficient data to provide reliable estimates of weights for 
commonly used predictors) » there is reason to believe that workable 
solutions to the problems involved in combining predictors may be 
found in approaches involving pooling data for several departments 
within the same field* The basic rationale underlying approaches 
involving pooling data for several small samples in a given field 
(sayt chemistry) is that there are substantial elements of similarity 



* Bec^iuse of small sample sizst multiple regression analysis was 
not employed routinely in the Cooperative Studies > and multiple 
correlation coefficients were reported, in selected studies > primarily 
to facilitate discussion of the principles and problems involved in 
developing and evaluating the predictive value of weighted composites 
of predictors (cf». Appendix C-1 pp# 7-8^ and 16-17; see also 
Appendix Tables 1-1 1)* 
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^ In th€ s^ntral typ«« of tasks psrfonsd by studsnts fro* dspsrtnsnt 

to departneat within the same field** 

Testing a "Common Weights" Hyp othesis 

Given data consisting of a common set of predictors and a 
comparable criterion variable for each of several departmental 
samples in a given field of study (say, chemistry), it is reasonable 
to ask whether (regression) weights for predictors as estimated from 
individual departmental data differ significantly from weights that 
may be estimated by the use of pooled data for all ttie departments 
involved.** 



* In a GRE-Board sponsored study, Boldt (1975; cf.. Appendix A) 
called attention to the importance of assuming significant elements 
of similarity in the types of activities involved in educational or 
occupational "pursuits of the same kind that are being carried out in 
different locations. Even though the tasks involved in a first-year 
chemistry program, for example, may be conducted at different levels 
of difficulty and with differing emphases from department to depart- 
ment, it is reasonable to assume that the general underlying 
similarities are at least ds great as the differences in tasks. 

** In graduate level validation research, and in other. validity study 
settings as well, questions may be raised regarding the practical 
utility and relevance for admissions decisions of gains, if any, that 
may accrue from differential weighting of a common set of predictors 
for each of several' different but similar prediction contexts. It 
may be argued that the level of precision Implied by "unique weighting 
is not justified considering (a) the presumed multidimensional nature 
of the assessment process that culminates in admissions decisions, 
(b) the fluctuations that occur from year to year and sample to sample 
in the magnitudes— even the signs— ot weights generated by within-groups 
analyses, (c) the limitations of performance criteria employed in studies, 
lack of a working consensus regarding one criterion as being the most 
appropriate or representative, and the presumption of a positive correla- 
tional manifold among all potential criterion variables, (d) the probably 
high degree of similarity across "similar" settings in the relative 
demands placed on general verbal and quantitative abilities, and 
(e) the typically high correlation between "reasonably" weighted 
composites of predictors. Several liberal arts colleges, law schools, 
or graduate chemistry departments, respectively, are likfely to have at 
least as many elements i^f similarity as of difference in their patterns 
of demands upon student verbal and/or quantitative abilities. It is not 
unreasonable to hypothesize that a limited number of sets of weights 
for a common set of predictors should be sufficient for purposes of 
within-group prediction in identifiable clusters of similar selection 
settings. See Wainer (1976, 1978) for an examination of the "weight- 
fluctuation" problem and rationales for dealing with It. 
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, Evidence that predictor tights estimated from individual 
departmental data do not differ significantly from weights estimated 
f rongi data pooled from all departments would support that Important 
underlying as sump t ion » regarding general similarities across departments 
in the same field* In addition, it would suggest that a single . 
**8olution'* to the problem of combining predictors might be applicable 
in each of several departments in a given field. One Important 
practical Implication is that each of the departments could use the 
pooled-departijient weights in developing a composite-predictor with 
local relevance, validity, and utility* 



Data and Analytic Approach* 

Data for 54 departmental samples from five fields vere selected 
for the exploratory analysis: biosciences including departments of 
zoology^ botany^ forestry* natural resources* marine science* 
general biology* chemistry* psychology* English* and history* The 
samples were from 25 different graduate schools* Table 14 shows for 
each of the five fields the number of samples invplved in the 
analyses* and the median and range of the sample sizes* 

Note in Table 14 that there were fewer samples with GRE-^Verbal* 
GRE-Quantitative* and UGPA (V*Q*U)* than with Verbal and Quantita^ 
tive (V*Q) only* Since scores on the GRE Aptitude Test <Verbal or 
V* Quantitative or Q) constitute the most commonly 'available set 
of predictors* it was considered desirable to test the common 
weights hypothesis for V*Q as the independent variables (called the 
V*Q analysis)* and then conduct a second series of analyses involving 
V*Q* and UGPA (called the V*Q*U analysis) by using data for a reduced 
number of samples* 

The analytical approach employed in testing the common weights 
hypothesis is outlined below* assuming the availability of three 
predictors (V*Q* and UGPA) and a **common" criterion* namely* 
Graduate GPA for several departmental samples* (It is important to 
note that the first step In the analytic procedure described below 
Is to standardize graduate GPA within each department prior to ' 
pooling data in order to control for differences in the grading 
scales) : 

Let us consider only one type of department at a time— «ay* 
chemistry* For the 1th school let Y. denote the graduate GPA 
In that school* Y Is a variable defined on all students in the 
given department In school 1* . 



* The consultative assistance of Paul Holland, office of Data 
Analysis and Research at ETS, who suggested the analytic approach 
used In testing the common-weights hypothesis* is acknowlt»dged with 
appreciation* 



Table 14 



Numl?er of Samples (Oe -tnents) in V»Q and V»Q»U 
Analysis > and Data on Size of the Samples 



No. of departments 
per analysis 



Biosciences 



Chemistry 



Psychology English 



History 



No. of V,Q samples* 
Median N 
Range of N*s 
V 

No. of V,Q,U samples** 
Median N 
Range of N*8 



(19) 
25 
(6-43) 

(13) 
28 
(6 - 43) 



(9) 

29 

(10 - 93) 

(6) 
19 

<n - 92) 



(12) 
40 

(20 - 89) 

(8) 
38 



(6) 

36 

(19 - 54) 

(5) - 
34 



(22 - 89) (14 - 51) 



(8) 
30 

(25 - 48) 

(7) 
29 

(25 - 48) 



♦Number of samples In analyse involving Verbal and quantitative scores (V,Q) as independent 
variables. • 

**Sun«ber of samples in analyses involving Verbal, Quantitative, and UCPA (V,4,U) is Independent 
variables. 
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1) Standardize Y to have mean 0 «nd variance I witliin school 1« 
^Thls 1b done because tne schools may have different grading systems 
we are pooling them in a regression so we want to remove this source 

. of between-school dlf ferences^ at least superf icially« 

2) Using the data from all students with complete records 
from all schools of the given department type^ estimate equations of 
the form: 



<1) « + b^V + b^Q ^• b^UGPA ' 1 « l,*.\,n. 

Note that this estimates common weights for Q» and UGPA across all 
schools but alLowa each school to have a separate Intercept term» a^« 

3) Now the question naturally arises: are there sufficient 
data from the 1th school to determine that it has weights ^t^f 
b^^, or b^^ that are different from the pooled weights? This is, 
done for by fitting equations of the form: 



(-i) Yj « + b^jV + b2Q f b^UGPA 

'^ ^ A A A 

Yj « aj + b^V + b2Q + b^UGPA for J/1* 



The actu.il fitting of these equations will be done by least squares 
and will use Indicator variables and their products with and 
UGPA to fit equations like (1) and (2)» 

4) Thf test for whether or not a soparate wui{;ht Is neodud for 
V in school I Is the l^doproo-of-f reodom F-test obtained by comparing 
the residual sums of squares from (I) and (2) in the usucil way« 



. Regre s sion R osults Wh en Data Were Pooled 

Following; the forogolu}; analytical approach, pooled departmental 
data won* xiHod to estimate n^gresslon weights and multiple correlation 
coefficients (a) for V and Q In one series of analyses, and (b) for 
V,Q, atid Underj^raduate GPA in a seccnul* Table 15 summarizes the 
pi>oJed wl th In-dopa rtment rej^resslon results for the V,(J and V^Q^U 
analyses, respett Ively , for eatMi ol the five fields* Thp wel{;hts 
shown r<»pr «vuMit estimates oi'. wei};hts ft>r >*j t an da r d j zod predletur 
scores, an»l the mtiltlple correlatioti yiel<led by Iho combined predle- 
tors based on all the avillable data* Several features of the 
findings are noteworthy, inrludlnp, tho following:. 
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Table 15 



Results of Pooled Departmental Data 
Regression Analyses* by Field 



Field 



Biosciences 



Chemistry 



PsycJ-ology 



English 



History 



No. of 

samples 

pooled 



Total 
no. of 
cases 



Standard regression weight 
GRE-V GRE-q UGPA 



Multiple 
correlation 
coefficient 



(19)* 
(13)** 

(9) 
(6) 

(12) 
(8) 

(6) 
(5) 

(8) 
(7) 



458 
390 

300 
203 

518 
326 

215 
151 

262 
228 



.177 
.178 

-.077 
.005 

.184 
.234 

.352 
.368 

.197 
.155 



.2C6 
.240 

.368 
.289 

.187 
.178 

.110 
.084 

.155 
.148 



.208 



.330 



.200 



.183 



.307 



.292 
.390 

.343 
.444 

.286 
.386 

.394 
.437 

.294 
.415 



♦Number of samples in analyses involving V and Q only as independent variables. 
**Number of samples in analyses involving V, Q. and UGPA as independent variables. 
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^ o Note that' combining GBiZ and UGPA acoras raaulta In 
Increaaed validity. The multiple correlation coef- 
ficlenta for V, Q, U are considerably greater than > 
those for V,Q only* This result is expected* 

• * 

o In one field* chemistry, 6RE-Verbal tends toward 
zero-weighting;* in English, Q makes a comparatively 
small contribution as compared with V, while in the 
remaining fields V and Q tend toward equal weighting* 

Testing deviations of departmental weights from the cooled 
estimates * Following the procedures outlined in the analytic 
approach, above, tests were made of differences between predictor 
.weights estimated by using data for individual departments and the 
predictor weights estimated by using the data for all the departments* 
Table 16 summarizes the outcome^ of tests* 

o In the analyses Involving V,Q as the independent variables 
or predictors, departmental weights for V were found to 
ievlate aignif Icant^y (p. ^ .05) from the. pooled 
' estimates in only six of 54 samples, across all fields, 
and deviant departmental weights for Q were indicated in 
only five of the 54 saiiq[)le8* 

o In the V,Q, U analyses, few sample weights were signifi- 
cantly deviant — in only two of ^ 39 samples, the weight for 
V was devlantCp* < .05); In only three of 39 sampJies, 
weights Q or UGPA differed significantly from the pooled 
estimates. 



yin evaluating the negative coefflclelit for GRE-V In this departmen- 
tal analysis. It Is Important to keep* in mind that when a negative 
regression weight Is obtained for an academic predictor, the predictor 
involved can be excluded from the set of predictors Involved In the 
analysis, in this case, all the Information of value for estimating 
first-year grades is being provided by GRE-Q (in the V,Q analyses), 
or GRE-Q and UGPA In the three-predictor analysis. Moreover, a 
negative regression weight may be obtained in circumstances In which 
the predictor Involved has a positive validity coefficient when 
considered separately. Consideration of this phenomenon, known as 
"suppression effect," Is outside the scope of this report. However, 
it is of considerable Importance In graduate-level validation 
research because It tends to occur under conditions that may be 
encountered in fields that are either heavily "quantitative" or 
heavily "verbal" (such as chemistry in this particular analysis) 
when both verbal and quantitative ability measures that are moderate- 
ly highly related are included in a prediction battery. [Of., 
discussion of the problem involved using a GRE-Aptitude total score. 
Appendix C-1, pp. 7-8; see also Wilson (1974) for evidence of 
recurring suppressor effects in undergraduate settings.] 
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Sunu^ 4)f «^tC9aMi^f TMts of Differmces 
tetVMn Smplik^ai^ Pooled EstiMtas 
of elglitsj^r Predictors 



Field/ 

independent 
variables 



Biosciences 



Chemistry 



^Psychology 



English 



GR£«*VerUl« ♦ , 
No* of No. Of 

tests * ileviant 
weights* 



V,Q Al9) 
V,Q,U (13) 

V,Q ( 9) 
V,Q,U ( 6) 

V,Q (12) 
V,Q,U ( 8) 

V,Q ( 6) 
V,Q,U ( 5) 



1 

0. 

2 
0 

3 
1 

0 
0 



ta^'^Qusntitative _ 
No* .vr # Ho. of 
tests * deviant 



weights** 



Undergraduate CPA 
No. of No. of 
tests ^ deviant 

weights*** 



(1 
•(13 



( V) 

( 6) 

(12) 
( 8) 

( 6) 
( 5) 



0 
0 

2 
1 

0 
0 



Not applicable 
(13) 0 

Not applicable 
( 6) 0 

Not applicable 
( 8) 1 

Not applicable 

( 5) ' 0 



History 



V,Q ( 8) 
V.Q.U (-7) 



0 

r 



( 8/ 
• ( 7) 



1 
1 



Not^pplicable 
( 7) 2 



AU fields 



V.Q (54) 
V.Q.U (39) 



6 

2 



(54) 
(39) 



5 
3 



Not applicable 
(39) 3 



*ln V.Q analyses, the weight for V in a sample. e8y5gt£lJi g>"f t * common (pooled) estimate for _^. 

differs significantly from the pooled weight for V (p < .05); in V.Q.U analysesv welRhts for\Q 
and UGPA are constant In each test for d epartmental weight f or 'V.. 

**ln V.Q analyses, <hv weight for Q in a sample, MtJLimu^^^ 

differs significantly from thu pooled wt-ight for Q (p ? .05); 1» V,Q,U analyses, weijj.lit8 Jor^V 
and UGPA are constant in each test for departmental we l>;ht for Q . * 

***ln these analyses, the wvight for U in a sample* »L«Liait<li Jn „ 
sain£lo weights for V a nd Q, differs significantly trom the pooled estimate of the weight for U 

Vp Vd5) . 
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o Mo ntCMitie 41r«etloii in!iAieAe«4 for tho toot* ' 

reoulto. Th*| io, in soven oaaploo in which 
wolghto for *ono or ooro prodtctoro i«oro Idonti- . 
fiod as deviant, the samplo voight was larger 
than the pooled estlnuite, while in eight casea 
* the sanplo weight wao taaller (and in aone > * . 
inatancea, anonalouaiyi negative)* 

• \ ♦ • 

Examination of aamnlea with Meviarit weichty . The test results 
indicate that the data generally (Conform to' ^he "conaon weigHta" 
hypothesis* However, what about the deviant departments? Are there 
characteristics in the data that may help to explain -the "deviant"^ 
outcomes? To shed light on these questions, a detailed examination 
was made of the data in all departments in which one or more predict- 
tors were identified as having regression weights differing aignifl- • 
cantly from the pooled estimate* ' * 

In almost every instance, examination of the original data for the 
13 samples involved (with Ns tanging from 11 to 52) revealed conditioqs 
that help to account for "deviant" regression weights* Detailed results 
of the examination are outlined in Appendix D* However, the essential 
nature of the findings may be sunmariaed as follows: 

o In samples characterized by atypically high positive 
regression weights for 6RE-V and/or 6RB-Q, the observed 
result was associated with one or more atypical data sets 
for individuals who were identifiable in certain ways as 
"atypical"— ^*g., members of minority groups with very low 
test score (s) and also very low graduate grades* *tSee 
detailed departmental analyses in Appendix D*} 

♦ * * 

♦ 

o In samples with anomalous negative coef f icients^ outcomes 
%rere clearly associated with one or two extremely atypical 
data-sets or outliers that heavily influenced results^^ogo » 
one individual with* unusually low standing on a predictor 
and unusually high standing on the criterion^ or vice versso* 



* Careful examination of the detailed data in Appendix. D will 
reinforce this important point regarding the Impact that one cr two 
aberrational data ^ets> or outliecs» can have on the magitiitude 
and/or the sign of validity coefficients in small samples* Negative 
coefficients, of course, are anomalous— -ioe* , coefficients reflecting 
the relationship of academic predictors (such as GRE scores) to 
academic criteria (such as grades) should be positive, a priorlo 
Given the potential for anomalous outlier impact, the overwhelmingly 
positive distribution of validity coefficients obtained in the 
Cooperative Studies in data for very small departmental samples 
indicates a remarkable degree of underlying regularity in such 
datao Attention to sample definition, however, clearly is necessary 
in order to avoid confounding resultso 
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♦ IttDlicatlons of the Findings 



On balance^ the findings d£ 'these exploratory analyses lend « 
support to the common weights hj^thesis and to the important 
assumptions under lyittg the use of pooled tlata for several small., 
departments in 6RE validation research^ the following points are 
important: 

o Results of the regression analysis » per set clearly 

indicate that prediction should be Improved by an appro-* 
priate combination of available GRE and U6PA predictors^ 

o It may safely be assumed that small departments are not 
(and are never likely to be) in a position to obtain 
reliable estimates of predictor weights by using local 
data only* 



o Results of the series of te8ts» on balancet support 

the common weights hypothesis* They enhance the prospect 
that pooled--data analysis may provide solutions to the 
problem of combining predictors that can be applied in 
local departmental settings ♦ 

o The findings point up the importance (a) of validity study 
models employing data that are generally comparable across 
departments » and Xb) of the concurrent participation of 
several departments from the same field in cooperative 
validity studies* 

It is important to recognize^ in connection ^ith the foregoing 
points, that it is not necessary to hypothesize a strictly "common 
weights" solution to the problem of estimating predictor weights, 
using pooled data approaches, that may have local applicability, 
validity, and relevance for several graduate departments in a 
field* So-called Bayesian methods of analysis have shcnm promise as 
a means of "adjusting" locally derived regression weignts on the 
basis of findings in aggregated or pooled samples (e*g*, Boldt, 
1975; Rubin 1978)* These methods have been applied successfully in 
contexts involving relatively large "local" samples (e*g*, in 
undergraduate and law school settings)* The important consideration 
is that pooled data approaches that have had demonstrated effectiven 
in certain settings appear to offer spticial promise for «radat to- 
level validation research* 



The present exploratory study represents a useful first step* 
It should be kept in mind that the departmental samples involved are 
not necessarily representative of all graduate departments from 
their respective "fields*" Further empirical study is needed and 
appears to be fully warranted on the strength of the findings that 
have been reviewed* 
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For a variety of reasons » assessment of the predictive validity 
of GRE tests In graduate school settings has not been carried out oji 
a regular ba^ls* The volume of validity studies Involving GRE tests 
lias been low relative to the number of settings in which validity 
studies could and should be conducted and as compared with the volume 
of studfies conducted In iindergraduate and certain professional 
school settings %ihere circumstances have been conducive to the 
widespread » r<Mitlne application of standard validity study models 
and procedures • 

In commissioning the GRE Cooperative Validity Studies Project » 
the Gradiiate Record Examinations Board accepted » at least tentatively » 
certain basic premises » as follows: 

1) In light of ' past experience^ and considering the Inherent 
complexities Involved In conducting graduate«-level validity 
studies » It seems iinllkely that concerned graduate schools 
and departments will be able to monitor GRE predictive 
validity thoroughly and regularly through self«»lnltlated 
studies alone; 

2) The participation of graduate schools and departments 
in the validity study process on a regular recurring 
basis » the generation and widespread dissemination of 
up-*to-date and interpretable information regarding GRE 

^ predictive validity in a variety of contexts^ and the 
Improvement of validity study procedures generally » are 
goals that are shared by all parties concerned with GRE 
development and use; 

3) Attainment of these shared goals is most likely to be 
realized through sustained cooperative interaction between 
all concerned parties^ namely^ graduate schools^ Educational 
Testing Service » and the Graduate Record Examinations 
Board* 

The Project was charged with developing and exploring the 
utility of one or more specific models for facilitating and encouraging 
the participation of graduate schools and departments in GRE validity 
studies • It was assumed that experience gained during the project 
would contribute to the development of arrangements and procedures 
through which the GRE Program might facilitate the regular participation 
of graduate schools and departments in validity studies* 

The activities involved and the findings generated in carrying 
out this charge > over a three--year period^ with the sustained 
support and encouragement of the Graduate Record Examinations Board > 
have been described in detail. The graduate school community was 
invited to participate in cooperative studies based on a sharply 
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focusttd* highly structured validity study model, with limited, 
clearly defined procedural and data requirements; ETS offered to 
conduct studies for and report findings directly to each graduate 
school willing and able to provide the needed data for one or more 
departmental samples, at no cost to the participating school* It 
was reasoned that data generated by this procedure could be compared 
across departments within Institutions and would facilitate the 
comparison of findings and the assessment of trends within and 
across fields. 

This approach was successful In enlisting the cooperation 
of 39 graduate schools, represented by from one to 17 departments 
per institution. In validity studies. Individualized Institutional 
reports were prepared for each participating school. More than 
150 data-sets, generally corresponding to departments and represent- 
ing over 19 fields or clusters of fields, were analyzed. The data 
generated by these studies permitted the analysis of trends In 
patterns of correlational validity for GRE Aptitude and Advanced 
tests and Undergraduate CPA In recently enrolled cohorts of first- 
time, full-time graduate students, primarily with respect to first- 
year Graduate CPA criteria. A report summarizing the findings of all 
Institutional studies was sent to each participant In the Project. 

The findings indicate that the frequently cited problems of 
conducting graduate-level validity studies are very real. Problems 
associated, with small samples, unstable weighting, restriction of 
range, criterion selection, and so on, are inherent in graduate 
school settings and must be dealt with In all graduate-level validity 
studies. However, experience during the Project indicates quite 
clearly that it is possible to conduct basic validity studies 
yielding useful, interpretable results despite these problems. 

For analyses involving very small departmental samples to yield 
useful results, it Is important to make sure that the samples are 
clearly defined and relatively homogeneous with respect to student 
educational status at entry, and that students are engaged In 
comparable pursuits over a defined study period. It is believed 
that careful atttntion to the problem of sample definition contributed 
significantly to the generally Interpretable nature of the findings 
obtained in the Cooperative Studies. Only first-time, full-time, 
degree-seeking students were included In the samples (and the 
findings, of course, apply only to such samples). 

Small sample results become increasingly meaningful as data 
from several departments in each of a variety of fields can be 
aggregated to provide normative perspective for assessing trends 
In patterns and levels of correlational validity for several 
predictors within and across fields. 

Results of special analyses Indicate that i»«»ollng procedures, 
using data for comparable sets of predictor and criterion variables 
for several small samples In the same field, have considerable 
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prowlse a« a basis for arriving at reliable estimates of validity 
coefficients and weights for predlctors-veights that may be used by 
small graduate departments in a given field to combine available 
predictors in such a way as to form a composite-predictor havina 
local relevance and predictive validity* 

The findings of the Project provide firm support for the 
interpretive rationale posited at its inception, namely, that 
measures of developed ability and achievement (such as GRE scores 
and Undergraduate CPA) should tend to be positively related to 
measures of performance in graduate study (such as the Graduate 
GPA)* The overwhelmingly positive pattern of relationships found in 
this project add to evidence from earlier studies that GRE scores 
and undergraduate grades provide relevant information that can be 
-^Lo r""' . complex process of screening applicants for 
admission to graduate school. Limited evidence was also provided 
regarding the validity of GRE tests for women, minorities, and 
foreign students. However, analyses were based on very small 
samples. Special efforts will be needed in order to obtain more 

vaUdlty data for these and other subgroups of special 
interest (e.g., older students, part-time students). 

c believed that the results attained during the Cooperative 

Studies Project indicate the validity of the premises underlying 
commission of the project by the GRE Board. All parties to GRE 
development and use have a responsibility to develop current answers 
to questions regarding the predictive validity of GRE and other 
admissions variables— answers calling for empirical evidence regarding 
the relationship of these variables to clearly defined and relevant 
criteria of performance in representative cohorts, demographic 
subgroups, and graduate admissions settings. Answers to these 
questions must be kept current to monitor changes in validity that 
may occur with changes in student populations, graduate programs and 
curricula, grading standards, conditions of test use, and the 
characteristics of the GRE tests themselves. 

Procedural and other arrangements ace needed to facilitate the 
recurring participation of all GRE-using graduate s<th.. .Is and depart- 
ments In basic, standard validity studies. Recurring participation 
in studies is even more important in graduate school settings, with 
characteristically small departmental samples as the units of 
analysis, than in undergraduate and law school settings with large 
entering cohorts as the units of analysis. In these latter settings, 
the availability of program-supported admissions-related research 
and validity study services has been directly responsible for the 
development of regular patterns of institutional participation 
in validation research. 

Sustained cooperative arrangements Involving the GRE Board, 
Educational Testing Service, and concerned graduate schools can 
contribute directly to the regularlzatlon of the graduate-level 
validity study pro.tnss. Several features of existing program-related 
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follOMSX 

1. Primary focus on a clearly defined study period (typically. 
^h^fir«t year of study), and a basic, limited core of 
iaUduJ a^^J fata (typically, test scores, a measure of 
^^dergraduate performance, and a criterion measure), with 
^e Options for extending standard studies on an ad hoc 
basis; 

2. Maximum use of the program's central data file to facilitate 
' ^ cSlection of validity study data and ^tension of 

services (descriptive statistics etc) to test-users, 

3. Regular, publicized cycles of participation; 

A. NO cost to participating institutions for a»*ly»i« *f 
reporting, with funding on a programmatic, continuing 
basis. 

Plana for a continuing GRE Validity Study Service f»^odylng 

those outlined above have been developed by the 

graduate school organization will not be easy. However, i" develop 
Se^ offers an exciting and challenging opportunity fo'^/^^ji^^^f . 

of GRE scores for those decisions. 



5./ 



Refereaces 



Boldt, R. R., Ck>mparl8on of a Baveslan and a least squares method of 
educational prediction- (ETS RB 75-15) Princeton, NJt Educational 
Testing Service, 1975. 

Burns, R. L., Graduate admissions and fellowship selection policies 
and procedures^ Part 1 and Part 2 (Supplementary Tables) , 
Princeton, NJ: The Graduate Records Examination Board and 
Educational Testing Service, 1970* 

Carlson, A. B. , Evans, F. R;, & Kuykendall, N. J., The ffcaslblllty 
of common criterion validity studies of the GRE (ETS RM 73-16). 
Princeton, NJ: Educational Testing Service, 1973. 

Creager, J. A., A study of graduate fellowship applicants In terms 
of Ph.D. attainment (Technical Report No. 18). Washington, 
DCl Office of Scientific Personnel, National Academy of 
Sciences — National Research Council, 1961. 

Creager, J. A., Prev Ictlng doctorate attainment with GRE and other 
variables (Technical Report No. 25). Washington, DC: Office 
of Scientific Personnel, National Academy of Sciences — 
National Research Council, 1965. . 

» » 
GREB Newsletter (No. 20), Spring. 1975. , 

Harvey, P. R. , & Lannholm, G., V., The performance of foreign graduate 
students on the Graduate Record Examinations Aptitude Test (GRE 
Special* Report 61-1). Princeton, NJ: Educational Testing 
Service, 1961. 

Harvey, P. R. , & Pitcher, B.v The relationship of Graduate Record 

Examinations Aptitude Test scores and graduate school performance 
of foreign students in four American gradxiate schools (GRE Special 
Report 63-1). Princeton, NJ: Educational Testing Service, 1963. 

Lannholm, G. V., Abstracts of selected studies of the relationship 
between scores on the Graduate Record Examinations and graduate 
school performance (GRE Special Report 60-3). Princeton, NJ: 
Educational Testing Service, 1960. 

Lannholm, G, V. , Review of studies employing GRE scores In predicting 
success in graduate study , 1952-1967 (GRE Special Report 68-1). 
Princeton, NJ: Educational Testing Service, 1968. 

Lannholm, G, V., Sumroarleg of GRE validity studies, 1966-1970 (GRE 
Special Report 72-1). Princeton, NJ: Educational Testing 
Service, 1972, , 

\ 



6 i 



-54- 

« • * - • 

References (contM*) 

Lannholm, G. V. , & Schrader, W. B., Predicting graduate school success— 

An evaluation of the effectiveness of the Graduate Record Examinations* 
Princeton, NJ: Educational Testing Service* 1951. 

Lannholm, G. V. , Marco,, G. L., & Schrader, W. B. , Cooperative studies of 
predicting" graduate school success (GRE Special Report 68-3). 
Princeton, NJ: Educational Testing Service, 1968. 

« 

Linn, R. L. , Fair test use in selection. Review of Educational Research ^ 
1^73, 43, 139-161. 

/ ■ • 

Linn, R. L., Test bias and the prediction of grades In law school. 

Journal of Legal Education, 1975, 27,, 293-323. 

National Academy of Sciences — National Research Council. Doctorate ' 

recipients from United States universities . 1958-1966 (NAS ' 
Publication- 1489). Washington, DC: NAS~NRC, 1967. 

Rellly, R. R., Critical incidents of graduate student performance 

(GREB Technical Memorandum No. 1). Princeton, NJ: Educational 
Testing Service, 1971. 

Rellly, R. R. , Factors in graduate student jperformance (ETS RB 74-2) . 
Princeton, NJ:* Educational Testing Service, 1974.- 

t 

Rock, D. A. , & Harmon, L. R. , The identification of papulation 
moderators and their effect on the jpredlctlon of doctorate 
attainment . Unpublished manuscript. Educational Testing 
Service, 1972. (a) » 

Rock) D. A/, & Harmon, L. R» , The prediction of doctorate attainment 

In psychology^ mathematics, and chemistry (GREF Preliminary Report) « 
Princeton, NJ: Fducatlonal Testing Service, 1972* (b) 

Rubin, D. B. , Using empirical Bayes techniques In the law school validity 
studies. Report #LSAC 78-la. Princeton, NJ: Law School Admission 
Council, 1978. 

Schrader, W. B. , The predictive validity of College Board admissions 
tests. In Angoff, W. H. (ed.). The College Board Admissions 
Testing Program ^ Princeton, NJ: College Entrance Examination 
Board, 1971, 117-146* 

Schrader, W. B., Summary of law school validity studies, 1948-1975, 
Report '//I4SAC 76-8. In Law School Admission Council, Reports of 
LSAC Sponsored Research: Volume III, 1975-77 . Princeton, NJ: 
Law School Admissions Council, 1977, 519-550. 



* References (cont'd.) 



Schrader, W. B, , & Pitcher, B. , Discriminant function analysis In the 

prediction of lav school performance (EIS Statistical Report 7^2-40). " 
Princeton, NJ: Educational Testing Ser.vlce, 1972. 

Sharon, A. T. , Test of English as a foreign language as a moderator of 
Graduate Record Examinations scores In the prediction of foreign 
students' grades In graduate school (ETS RB 71"50). Princeton, 
NJf Educational Testing Service, 1963. 

Tucker, A., Gottlieb, D. , & 9ease, J., Attrition of graduate students at 
the Ph»D. level in the traditional arts and sciences (Publication , 
No. 8). East Lansing, MI: Michigan State University, Office of 
Research and Development and the Graduate School, 196A. 

Walner, H. , Estimating coefficients in linear models: It don't make 
no nevermind. Psychological Bulletin , 1976, B3_t 213-217. 

Walner; H. , On the sensitivity of regression and'regressors. Psychological 
Bulletin, 1978, 85, 267-273. 

Warren, J. R. , Prediction of college achievement among Mexican-American 
students in California (ETS RDR 76-77 & RB 76-22). Princeton, NJ: . 
' Educational Testing Service, 1978. 

Wlllinghara, W. W. , Predicting success in graduate education, > Science , 
1974, 183, 273-278. 

Wlllingham, W. W. , Predicting success in graduate education . Papers 
presented at the Graduate Recbj^d Examinations Board Research 
Seminar, 12th Annual Meeting o^ the Council of Graduate Schools. 
Princeton, NJ: Educational Testing Service, 1973. ^ 

Wilson, K. M. , Predicting the long-term performance in college of 
minority and nonrolnorlty students: A 'comparative analysis in 
two collegiate settings (ETS RDR 77-78, No. 3 & RB 78-6). 
Princeton, NJ: Educational Testing Service, 1978. 

Wilson, K. M. , Of time and the doctorate— Report of an inquiry into 
the duration of doctoral study . Research Monograph No. 9. 
Atlanta, GA: The Southern Regional Education Board, 1965. 

Wilson, K. M. , The utility of a standard composite for forecastij^g 

academic performance in several liberal arts colleges. Research 
in Higher Education , 1976, 5, 19J-213. 

Wilson, K. M. , The contribution of measures of aptitude (SAT) and 
achievement (CEEB Achievement Test Average), respectively, in 
forecasting college grades in several liberal arts colleges 
CKTS RB 7A-36). Princeton, N.I : l-V.ucat lonal Testing Service, 1974. 



-57- 



Appendlx A 

GRE Program Related Effbrts to Promote Validation 
Research: Review and Appraisal 



Appendix A (Page 1 of lA pages) 



GRE PROGRAM-REUTED EFFORTS TO PROMOTE VALIDATION RESEARCH: 

REVIEW AND APPRAISAL 

During the past quarter of a century, those concerned with 
the development and use of the Graduate Record Examinations (GRE) 
have periodically called attention to the need for reliable knowledge 
bearing on the validity of the GRE tests in the selection of Indivi- 
duals for admission to graduate programs. The need to improve the 
validity study process in graduate schools has been endorsed and 
supported* 

A number of projects undertaken by ETS (with the encourage- 
Wnt of either the Committee on Testing of the Association of 
Graduate Schools (AGSl or, more recently, the Graduate Record 
Examinations Board (GREBl ) have been designed (a) to improve the 
quality of information available regarding the validity of the GRE • 
tests, (b) to advance understanding of the validity study process, 
and/or (c) to focus attention on special problems or promising 
development^. The projects undertaken to date have been of three 
types: 

a) those concerned with periodically collecting and dissemi- 
nating information and insights gained from institutional, 
departmental, and other validity studies, 

b) those concerned with designing and conducting validity 
studies with the cooperation of individual institutions 
or departments, and 

• 

c) those concerned with the study of particular applied, 
methodological, or conceptual aspects of the validation 
process or with particular problems and issues. 

A review of these three types of effort provides useful 
perspective for developing a strategy for Improving the validity 
study process (%»hlch has not become an established aspect of 
institutional operations at the graduate level). 



Collecting and Disseminating Information 

In the first summary of information on institutional or 
departmental validity studies, Lannholm and Schrader (1951)* described 
major studies of the prediction of graduate school success by the 



*See consolidated references following main body of the report 
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6RE tests that vers reported to the ORE Office during the period 
1937 through 1951* The investigators concluded that carefully 
constructed tests of achievement In major subject matter fields may 
be used effectively In the admission and guidance of graduate 
students and that the' Advanced Tests might be given precedence over 
the then available Profile Tests for purposes of predicting success. 

The next summary of validity studies was not forthcoming 
until November 1960, when Lannholm (1960) summarized results of 
a limited number of validity studies In a report designed (a) 
"to illustrate different approaches to a study of [the relationship 
between scores on the 6RE and success In graduate study] and (b) 
to stimulate other graduate schools to design and carry out studies 
of their own*'* Authors of unpublished studies were Invited to 
send copies of their reports to ETS* 

A more comprehensive summarisation appeared eight years later* 
Lannholm (1968) reported on 36 studies conducted over a 15 year 
period— i*e*, between 1952 and 1967— thirteen of i^ich were in 
the field of Education* In examining the various approaches taken, 
Lannholm noted that most of them involved the analysis of data 
separately for samples by discipline pr department but that a few . 
studies pooled data for samples from several disciplines or depart- 
ments* The only study included in this summary «hich involved 
the pooling of data from more than one graduate school was a study 
by Creager (1965) of the relationship between GRE scores and 
several related doctorate-attaiiment criteria in a national sample 
of applicants for NSF fellowships* 

In the 1968 report, attention was focused squarely on the 
"persistence of certain problems" in connection with designing 
and conducting validity studies in graduate schools* Lannholm 
cited as the principal problems those related to (1) the small size 
of samples, (2) the lack of a single satisfactory index of the 
effectiveiK'SS of predictors, and (3) the limitations of measures 
of gradual^ school success* The perceived need to conduct validity 
studies by department contributed to restriction of sample size; 
difficulties involved in interpreting correlation coefficients 
in highly restricted ranges of talent were held to militate against 
the routine use and interpretation of familiar correlational 
procedures for assessing predictor effectiveness; and the limita- 
tions of grade point averages, frequently employed as criterio^ 
measures, were cited* Lannholm characterized as "**.both surprising 
and disappointing***" the failure of most Investigations to Include 
a measure of undergraduate performance (e*g*, undergraduate grade 
point average) as a predictor* 

In the most recent summary report^ Lannholm (1972) presented 
the results of 14 studies, received by the GRE Office after the 
1968 report had been prepared, for the period 1966-1970e 
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AsAin, •mphaslt pl*c«d on coat limed problens o£ Mopls 
slse.* Effort* to enlarge settle, else by pooUng date for two 
or aore departnenti were noted* However, Unahpla noted that 
even though "...larger nunbere reaidt from pooling data froa 
different departnenta, the effect upon the prediction coeffldenta 
is difficult to determine, eapedally idien the abUltlea raqvlred ^ 
and tl^ perf omance atandarde vary from one department to *aother. 
An undergraduate grade point average waa uaed in conjunction with 
teat acorea In ten of the 14 atudlea; It waa auggeeted that adjuat- 
nenta for quality of the undergraduate Inatltutlon aeaned promlalng. 
The need for further work on the development of aatlafactary criteria 
of aucceaa In graduate atudy waa atreaaed. 

No comparable aummarlaatlon of the reaulta of departmental 
validity atudlea haa been reported since 1972. However, InterMt In 
this line of endeavpr contlnuea; the Spring 1975 laaue of the GRE 
Board Newaletter Included a request that graduate achoola forward 
reporta of validity atudlea carried out within the paat five yeara. 

The collection and dissemination of data on validity provided 
by local, Inatltutlonal and departmental studies clearly constitutes 
a necessary element In a comprehensive plan for Improving the 
validity study process, iiowever, the fact that this approach la not 
sufficient has been recognized; other approaches that have been 
supported are considered In the following sections. 



Cooperative Validity Studies 

In 1962, recognizing the limitations of many Institutional- 
departmental validity studies, the Committee on Tests of the AGS 
recommended that ETS undertake validity studies In cooperation 
with several graduate schools. In 1963, 32 departments In 15 
different universities were Invited to participate In studies 
designed to (a) evaluate the effectiveness of scores on the GRE 
and other factors In predicting success In graduate study In 
certain departments In selected graduate schools, and (b) to 
provide suggestions to other departments and schools that might 
wish to study the effectiveness of their own selection procedures 
(Lannholffl, Marco, and Schrader, 1968). 

The "cooperative validity studies" approach represented an 
important advance. Local studies were seldom strictly comparable 
in design and methodology and they typically employed different 
kinds of criterion meesures and samples. The cooperative validity 
acudlM moiel, on the other hand, employed a standard methodology 
and design. Analyses were centrally planned and conducted, while 
the departments cooperated in supplying the necessary data. 

k 



• 
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D«t« wtre received on saaples of etudente from 21 'depert- 
meats, spanning six fields st ten universities* The studies 
were conducted on a depertnental basis; the design did not call . 
for pooling data ftom different departaents within the sime discipline* 
• Comparable sets of predictor-criterion data were generated for^nany ' * 
of the departments, however* 

♦ • « 

These studies- used a complex criterion variable' defined In . « 

such a way as to reflect (a) the **progress** of individuals through 
various aspects of the "general excellence** of students* **Progres8 
during the study period** (and status at the time of the cut-off date 
Involved) was reported for each student as follows: 

a* earned Ph*D* 

b* passed all examinations, still enrolled 

c* has not passed all examinations, still enrolled 

d. withdrew at the request of the university (dismissed) 

e. voluntarily withdrew after more than a /ear of study 
f * voluntarily withdrew after less than a year of study 

Departmental ratings were employed along with the foregoing categories 
to define ^successful** and ^'unsuccessful** groups as follows s 

^ 'successful" students were those who had received the degree 

or who» If still enrolled, had highest ratings of ''outstanding** 

or "superior ^ 

"unsuccessful" students we *e those who had, not received the 
degree, who were still enrolled with "average or lower ratings, 
or who had .withdrawn regardless of circumstances e 

Only two of the samples studied Included more than 85 Individuals; 
the small samples no doubt contributed to the variation In results 
from one group to another ♦ It was evident from the study, not only 
that the validity of the GRE and undergraduate predictors varied 
considerably,* but also that there was marked variability among the 
departments %rlth respect to the distribution of students accoifdlng 



*In [retrospect, several factors may have contributed to variation 
^^ji^es^lts from one sample to another and have had an attenuating 
effect on the validity coefficients obtained* Among these factors 
are the classification as "unsuccessful" of students who withdrew 
voluntarily and the inability, due to small sample size, to analyze 
data separately by sexe * The number of males and females involved 
was not reported e However, the criterion Involved had as one of its 
elements "degree attainment within a specified time periods" Women 
have tended to take longer to complete decree requirements due to a 
number of non^ablllty sex-role*linked factors « Analyses by sex are 
important e 
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to status at tha and of tha study parlod Tabla In soaa 



Insart Tabla A-1 about hara 



dapartnants with raspact to tha distribution of studants according 
to status at tha and of tha study parlod (saa Tabla !)• Ixi sona 
dapartaants,* for axampla, alaost two-thirds of tha sanpla had ** 
attainad tha Fh.O*; in othars froa 40 to 94 parcant had not.passad 
all aicaalnations; in still othars, half or sora of tha studants had 
aithar withdrawn voluntarily or baan disaissad** 

Lannholtf, Harco, and Schradar (1968, p* 84) condudad that 
"on tha wfaola, tha rasults of thasa studiaa aaka It daar that - 
prad4etion of succass in graduata work, is axcaadin^ly difficult...* 
-Slid that in viaw of tha critical iaportanca of graduata study 
and jCha iaportanca of affactiva pradiction both to tha studant 
and/to tha j^aduata school, tha rasults taphasisa tha urgancy 
of/saaking ways to iaprova pradiction*** 

^ praviously no tad, tha 1968 CooparatiVa Validity Studiaa 
(launchad in fall, 1963) aabodiad tha principla of applying a 
standard study dasign to data for aach of savaral dapartaants* . 
ETS' invastigators wtra rasponsibla for study dasign, whila tha 
dapartaants wara askad to cooparata by supplying spacif iad data for 
tha study; and tha invaatigators ware succassful in obtaining tha 
cooperation of 10 of tha IS schools and 21 of the' 32 dapartaai^ts 
originally invited to supply data for tha study* Good cooperation 
was obtained froa faculty aaabars in participating departaants in 
supplying ratings of students* 

A second graduate-level project calling for the cooperation 
of departaents in supplying data for a centrally designed validity- 
related ^tludy, was initiated with GREB sponsorship in 1970 (Boldt, 
1975)* This study was designed to examine the utility of '•special 
" new statistical techniques" (Bayesian analysis) for weighting a 
coaaon set of predictors in several prediction contexts where saall 
saaple size tands to be a problem. Eighty-one departments of 
psychology an4^54 departments of economics were approached* Despite 
the fact that this study was endorsed by the GREB Chairman through a 
covering letter to the graduate deans* the research could not be 
carried out due to the fact that most of the invited departments did 
not provide the required data (Boldt, 1975, pp* 12-14)* The few 
which did supply data had limited samples* 

1 

*See Wilson (1965) for evidence of marked variability among depart- 
ments in rate of progress of students in completing doctoral programs* 
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I>«partMiiUl Dif£ttr«neM in PAtttrni «nd 
Ratfts of Progress la Ph.D. Study 



Dspartmsnt 



Itatxis M of study cut-off dste (in psresat) 



Still Enrollsd 



Ph.D. Exaos Exsas aot 
N conplfttsd pssssd passed 
XXX 



Dlsalssed 



Voluatarlly 
Wlthdrswa 



Later Earlier 
X X 



Chemistry A 
_B 
Total 



(116) 
20) 



Eaglish 



C 

E 
F 



Total 



History 



6 
H 
I 



Total 



Philosophy >J 
Physics 



K 
L 
M 



Total 



Psychology N 
0 
P 

Q 
R 

s 

T 

Total 



37.9 
65.0 



19.8 
5.0 



0.0 

040 



8.6 
0.0 



7.8 
10.0 



25.9 
20.0 



136) 



41.9 



17.6 



0.0 



7.4 



8.1 25.0 



98) 
81) 
54) 
32) 



25.5 
13.6 
31.5 
28.1 



41.9 
2.5 

11.1 
6.2 



8.2 
46.9 

7.4 
40.6 



6.1 
13.6 
0.0 
9.4 



2.0 
12.3 
29.6 

9.4 



16.3 
11.1 
20.4 
5.9 



2651 



23.4 



19.2 



23.8 



7.5 



11.7 



14.3 



66) 
40) 
28) 



12.1 
7.5 
28.5 



1.5 
55.0 
35.7 



33.3 
22.5 
0.0 



19.7 
5.0 
3.6 



13.8 
2.5 
10.7 



19.7 
7.5 
21.4 



134) 



14.2 



24.6 



23.1 



11.9 



9.7 



16.4 



421 



38.1 



39) 
38) 
32) 



43.6 
0.0 
65.6 



2.6 
0.0 



0.0 



0.0 



11.9 



16.7 



0.0 
94.7 
34.4 



0.0 
0.0 
0.0 



17.9 
2.6 
0.0 



15.4 
0.0 
0.0 



109) 



34.9 



9.2 



43.1 



0.0 



7.3 



5.5 



49) 
47) 
44) 
38) 
36) 
26) 
261 



16.3 
42.6 
20.4 
39.5 
63.9 
19.5 
50.0 



24.5 
4.3 
6.8 
'5.3 
16.7 
46.2 
42.3 



36.7 
19.1 
13. C 

7.9 
19.4 
19.2 

:^.8 



266) 



35.0 



18.0 



18.4 



All fields/ 
departments 



(952) 



29.9 



18.9 



20.0 



6.1 
4.3 

0.0 

CO 
3 3 
0.0 



16.3 
2.1 
18.2 
31.6 
0.0 
7.7 
3.8 



0.0 
27.7 
6.8 
15.8 
0.0 
3.8 
0.0 



7.9 



7.0 



12.0 



8.6 



10.5 



13.7 



Compiled from Lannholm, Marco, and Schrader (1968), based on the October, 
1963 status of students ".s. first enrolled between the fall of 1957 and June 
I960,.." 

** 

Row totals should equal 100 percent within limits of rounding. 
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The methodological and exploratory nature of this particular 
study may have proved to be a deterrent to cooperation* Other 
conslderationa ai iiell viay have contribute to the failure of 
institutions to provide data^ including^ for examplet the nature 
and availability of the data requested* The clerical burden 
involved and issues relating to the confidentiality of student 
data vere cited frequently* Other reasons included the lack 
of availability of data on Fh*0*^s and the lack of availability 
of GRE scores due to the fact that the score^requirement policy 
nas not actually enforced • 

In any events this particular ^^cooperative study^^ project 
failed to elicit the required cooperation of graduate departments in 
tvo fields of study* However t the methodological aspects of the 
investigation were completed successfully with data available in 
College Board^s Validity Study Service files at EtS for a number of 
freshman samples* And^ as will be considered later » the concepts 
underlying the design of the study have Important implications for 
development of the validity study process in graduate-study contexts* 

It is evident that the two projects. reviewed above were 
only partially successful in generating validity study data* An 
expanded cooperative validity-*study model citlling for the participa- 
tion of ihdividual or defined groups t of departments in centrally 
coordinated and facilitated validity studies might include provision 
for institutional«-developmental involvement in planning and designings 
as well as in providing data for the studies* In any event coopera- 
tion and ( ollaboration constitute necessary elements j.n any 
overall plan. for facilitating the development of validity studies in 
graduate school settings* 



Studies of Special Problems and Promising Developments 

As previously noted» the project by Boldt (1975) did not 
elicit the cooperation of graduate departments* However t in 
its design Boldt introduced a number of ideas that have Important 
Implications for the valldity«»8tudy process* In essence t Boldt 
focused attention on the need for approaches to the study of 
validity which assume that there are Important elements of similarity 
in the tasks required of individuals in each of several different 
prediction contexts— e*g*> several different graduate departments of 
chemist ry-^even though these tasks may be conducted at different 
levels > and with differing emphases* 

Boldt (I975t pp# l«-2) offers the following relevant observations: 

In some population segments > such as minority groups^ 
graduate students^ and possibly various occupational groups > 
one often cannot find enough people at a single place where an 
acceptable criterion exists to conduct a statistical study of 
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the predictive validity of selection iaetruMnta* or at least a 
study in whose results one can have confidence* It is nore 
connon to find soMll groups froa the population of interest 
Interspersed through a variety of locations » perfoming tasks 
chat seem reasonably similar* Evaluation of the perfonunces is 
nade with reference to the group at a location but without 
reference to the perlonnances outside that group*** Thus, 
the groups may differ from each other in terms of average 
performance or in the variation in performance » but these 
differences may not be Inferrable from the corresponding 
statistics calculated using quantitative evaluations of perform- 
ances made at each location* 

Uhere several schools are involved, one would want 
to incorporate the notion that they are more or less similar* 
One would certainly not want to proceed under the assumption 
that all schools are uniquely different, conceivably, and that 
no prior information [of value for facilitating the assignment 
of weights to predictors] is in existence* 

The assumption that several graduate departments in a given 
field of study probably are engaged In a basically similar enterprise 
suggests the possibility of improving validity studies through 
designs which call for the consideration of common sets of predictor- 
criterion data on individuals in each of several "similar" dep artments* 
This would result In a substantial enlargement of the data base 
available for analysis and enhance the generall^abilit^r of fundings* 

6RE scores constitute a set of predictors which may be thought 
as common from one prediction context to another* However, the ' 
problem of establishing the "credibility" of a criterion variable 
(or variables) with comparable meaning across several graduate 
departments is not so readily solved • The "criterion problem" h^s 
been the focus of two recent GREB-supported projects, namely, 
Rellley's (1971, 1974) critical-incidents study of graduate-student 
behaviors aimed at "defining empirically a set of criterion dimensions 
upon ^Ich graduate faculty base judgments of student performance," 
and an exploration by Carlson, Evans, and Kuykendall (1973) of the 
feasibility of developing validity studies of the GRE, based 
on a "common criterion •" 

Rellley's investigation was designed to identify aspects 
of student behavior that might help define criterion dimensions 
which graduate faculty members could use tti judging student perform- 
ance. Procedures such as those developed by Rellley clearly should 
be useful In exploratory validation research • They provide a basis 
for taxonomlc Investigations as well~e.g., for clustering departments 
In terms of the types of student behavior deemed most Important by 
the faculty* 
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Citing the concern of the 6RBB \Retearch Connlttee over the 
"pe«iclty of velldlty dete for the 6RE»** end the interest of 6RE 
Conttittees of Eicanlnert in validity studies, Carlson et el* 
(1973) undertook en exploratory investigation of the feasibility 
of developing for one or «ore fields "a neasurable criterion 
which would be generally acceptable to at least a large segment 
of that field," probably a common set of essay questions to be 
administered to students at the appropriate level in each of 
several different departments. Discussion of the possibilities 
of developing and using an essay-type measure with GRE Committees of 
Examiners in Philosophy, French, and Literature in English led to 
the conclusion that "the problems of such a study were insumountable 
and the procedure was rejected*" 

Reactions of the respective committees were varied* In 
Philosophy it was deemed feasible to obtain agreement on several 
essay questions, but the Committee doubted the adequacy of such 
questions as a criterion; they were unable to specify a task or 
set of tasks which they would find to be an acceptable criterion 
(though they felt that "rating scales offered real possibilities"). 
In French, interest was keen but efforts to Implement the idea 
were not successful— many of the department chairmen indicated 
concern over the operational problems posed by introducing a 
special examination* For the Literature in English Co^imlttee, 
"essay questions" were not acceptable as a criterion for graduate 
student performance in their field* They did express interest 
in "attainment of tenure in a 'good' department" as a criterion 
(implicitly, "quality level" of the institution in which graduates 
were finally "placed"); problems involved in implementing this idea 
were explored briefly but it was not pursued further* 

Although the standard-set-of-essay-questions approach to 
developing a common criterion was considered to be inappropriate, 
there was considerable interest in the possibilities involved 
in using rating procedures* Based on a survey of selected depart- 
ments in five fields, some A3 percent of responding departments 
reported regular use of some form of rating, typically at mastftr's 
or Ph*D* examination times* 

The investigators concluded from the survey results that 
a sufficient number of departments were employing rating procedures 
to warrant some preliminary studies based on existing rating data 
but cautioned that it would probably be desirable to develop a 
uniform set of rating procedures before using ratings as criterion 
measures . 

These explorations of the feasibility of using a common 
criterion yielded a negative conclusion only with regard to the 
feasibility of employing one particular form of "common criterion" — 
a common set of essay questions. The negative conclusion does not 
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apply to the Idea of a conmoii criterion^ per 8e> or the validity of 
an Ifl^ildt aaeunptlon underlying the proposed use of such* a crl«* 
terlon^ naiaelyt that the jtasks required of students In different 
departments vlthln a given field are sufficiently similar to warrant 
the use of a common criterion measure (or^ that It 'Is possible to 
Identify a subgroup of departments Which are by some acceptable 
means judged to be sufficiently similar In regard to demands made on 
students to warrant use of such a criterion measure)* 

An extension of GREB-^upported research reflecting concern 
over the criterion problem Is represented In the study by Rock 
(1972) > with the collaboration of Lindsay Harmon> Which used data 
obtained from the NSF Fellowshlir appl^jLcant records and the NRC 
Office of Scientific Personnel Doctorate Records File (DRF)* The 
study was designed to evaluate the validity of GSLE Aptitude and 

.Advanced Tests as predictors of Whether or not a candidate (In 
psychology » mathematics > or physics) attained the doctorate within 

. a given period of tlme> extending and elaborating a line of Inquiry 
associated with Creager (1961> 1965)* 

The GREB«»spon8ored study > like Creager ^s earlier studies* 
examined the validity of 6RE tests vs* Fh«D* attainment In a sample 
undifferentiated with respect to Institutional affiliation* but 
It also sought to determine whether there were particular subgroups 
within the fields under consideration for which the 6RE test might 
have varying degrees of predictive validity* Examples of such 
subgroups are **quallty level^^ of the graduate school* age at the 
beginning of study and sex* And the study provided* Incidentally* 
relevant Information bearing on the potential value of attainment 
vs* nonattalnment of the Fh*D* **wlthln a reasonable time** as 
administratively practical ^'common** criterion (or component In 
such a criterion) In a validation model having both %d.thln«-department 
and acros8«-department components* 

A rationale for use of the relatively crude Fh*D* attainment 
vs« nonattalnment criterion was offered by Rock and Harmon: 

The most desirable criterion* of course* would be some 
measure of achievement as a scientist* Aside from the logical 
difficulties in arriving at any sort of agreement as to what 
is a relevant measure of scientific achievement* %re are faced 
with the operational problem of time lapse which must occur 
before such data can be col^cted* 

An alternative criterion of a more Intermediate nature 
Is whether or not one has attained his or her doctorate 
within a reasonable period of time* Attainment of the doctorate 
Is appealing on logical grounds since* **lt Is one test of the 
effectiveness of the overall selection process* l*e*, the 
decision to admit a student to graduate education or to admit 
him to candidacy for a higher degree Implies an expectation 
that his formal graduate education will be completed* The 
attainment of the [doctorate degree] Is the primary - indicator 
that such an expectation has been fulfilled ***One criticism*** 
Is that doctorate attainment lacks sensitivity In the sense 
that It cannot take Into account the various qualitative 
levels of performance among individuals attaining the Fh#D« 



Although the latter criticism nay well he valid... (If this 
criterion] Is sufficiently lacking In sensitivity, this In turn 
will he reflected in the relative level of its predictahllity 
(pp. 1-2).* 

The Ph.D. attainment criterion thus ignores "levels" within 
that classification, and it ambiguously assigns all other individuals 
to a nonattalnment category. Some of these individuals will later 
become "attalne^." Accordingly, the criterion should be perceived 
as reflecting differences among individuals in rates and patterns of 
progress in completing programs of Ph.D. preparation . 

6RE scores have been found to be related, consistently if 
modestly, to criteria which reflect '*rate of progress** toward 
the degree. Differences in "ability" may partially account for 
observed differences in average degree-attainment times by institu- 
tional attendance and degree pattern (see Table A-2). Differences 
in degree-attainment rates by "quality level" of institutions may 
also be ability-related (see Table A-3), and differences among several 
departments within a given field with respect to average rates of 
student progress may be accounted for partially by differences in 
"quality of student input." Thus, rate of student progress in 
completing degree requirements appears to have considerable promise 
as one component in a "common criterion" variable reflecting the 
progress of individuals in completing requirements for graduate 
degrees, especially the Ph.D.** 



*Rellance on "raw" attainment vs. nonattalnment criteria is under- 
standable in sfudles involving, samples not identified with particular 
institutions/departments and liu available data base which does not 
permit the development of more refined criteria. In study designs 
which involve analyses both within * and ncross-instini tlons/depart- 
ments (field constant), more refined cilterlo. grcr.p-*n9' may be 
developed so as to reflect in some appropriate combination, for 
example, "degree attainment vs. nonattalnment," faculty ratings of 
the overall excellence of the work done by degree attalners, rate of 
progress in gaining admission to candidacy, failure to qualify, 
dismissal on grounds of Inadequate academic performance, etc. 

**In carefully controlled "rate of progress" designs, all individuals 
Involved should be at the same stage of preparation at the beginning 
of a study period In order to have the same amount of time In which 
to attain the degree or to reach any specific level of preparation 
(e*g*, completion of course requirements, admission to candidacy 
through qualifying examination, etc.)* Studies using Fh.D* attain- 
ment vs* "nonattalnment" typically have not adequately controlled 
the "equal time" variable. For example, in order to enlarge the 
sample, students enrolled during, say, a given three-year period, 
are Included, but a uniform cut-off date typically is employed* 
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Tabltt A-2 

Basic Institutional Attsndancs and Dagrae Pattarns 
for Ph.D. Kacipiants and Associatad Maasuras 
of Dagraa Attainmant Tlma * 



Institutional and dagraa 
pattern 




Index of time 


taken 


% 


Registered tine 
mean years 


AB to Ph.D , 
mean years 


Direct Ph.D* 


[ 20.0 ] 


[4.81 


[ 5.6J 


AB - Ph.D. 
AB ^ Ph.D. 


( 3.8) 
(16.2 ) 


4.7 
4.8 


5.4 
5.6 


Mafit:er*s-Ph.D. Same school 


[41.4 1 


[5.31 


[ 7.41 


AB - MA • Ph.D. 
AB |t MA - Ph.D. 


(12.8 ) 
( 28.6 ) 


5.4 
5.3 


7.3 
7.5 


MA«ter'a-Ph.D. different school 


[ 38.6 ] 


[6.11 


[11.01 


AB - MA y Ph.D. 
AB y MA y Ph.D. 


( 1-^.6) 
( 23.0 ) 


5.9 
6.3 


9.8 
11.8 



Data from NAS (1967) for U. S. doctorates (excluding foreign degree 
recipients) for 1966. The variations in mean attainment times shown 
here for all degree recipients t^ to hold for essentially all broad 
fields. As suggested in the NAS %blication (1967, o. 77), this as- 
sociation "...mav be caused bv different student abilities in the 
different [institutional attendance and degree] patterns, but no data, 
exist to verify this guess." 
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Table A-3 



StlActftd Data on tha Pareaataga of Individuals Attaining 
tha Ph.D. within a Civan Study Pariod by "Quality" 
of Inatitution/Dap^rtaaat and Ability Uvala- 



• 

• Data sourca and field 

• 


Percent attainina Ph.D. vithin 
study period by "quality level" 


Total 


"Higher" 
Z 


"Middle" 
X 


"Lower" 
Z 




Rock and Harmon {1972)^ 










. Psychology 


44 


36 


26 


40 


Mathenatlcs 


61 


40 


31 


54 


Chemlatry 


77 


69 


53 


74 


Tucker^ Gottlieba and Pease (1964) 










Physical sciences 


79 


65 


56 


70 


Biological sciences 


80 


67 


49 


71 


Social sciences 


68 


53 


44 


59 


Humanities 


57 


49 


33 


50 


(All fields) 


(70) 


(57) 


(46) 


(62> 


Creager (1965)*** 










Biology (males) 


62 


35 


30 


37 


Chemistry (males) 


63 


50 


25 


51 


Mathet^atics (males) 


37 


13 


0 


24 



Study period: 



Scudy period: 



*** 



Study period: 
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1958-61 through June, 1968; "quality" based on Cartter and 
other data descriptive of graduate department in which en- 
rolled 

1950-53 through December 1962; "quality" or productivity 
defined as (1) top 15 universities In Keniston rankings, 

(2) 300 plus Ph.D.s awarded, 1936-56, but not top 15, and 

(3) less than 300 Ph.D.s awarded and not top 15. 

This was a study Involving approximately 24,000 post-master 
students at 24 selected universities. The investigators 
concluded, in part, "...that to Increase Ph.D. production 
and reduce attrition, graduate schools would embark on active 
programs of recruiting potential graduate students and be 
mora selective in their admissions." (p. 293). 

1954-57 through August 1964: quantitative ability levels 
(GRE Q) defined as follows— "higher" - stanlnes 8-9; "middle" 
- stanlnes 5-7; "lower" • stanlnes 1-4. (Table 6, p. 24, 
selected fielda only, to illustrate trends). 
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Appendix B 
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B-l. Forms used In survey of deans of C6S-member 
graduate schools; covering letter form 6RE 
Board chairman ' • 



B-2. "Survey of Graduate School Validity Study 
Activities and Interests: Summary of 
Findings" 

A report of findings of the survey 
of deans ' 



B-3. Basic one-year, two-cohort vpiidity study 
model used ^or the Cooperative Studies 

Statement regarding confidentiality 
of treatment of data 

Study definitions, data collection 
procedures, etc. 



B-4. Brief description of selected studies using 
a two-year, single-cohort ^tudy model 

Study definitions, data collection 
procedures, etc. 
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Appendlst B-1 



t 



Graduate Record Examinations Board 



1IN A^f^lUlATlON WITN 



t V^MWv IttiOtt 

' AitrHif f J#ck»©« 
#a4 T«eM^«> ttM* M<MV«r«l|f 



C«^<t of OfAdutU S€i«M*» 
LOrt<»«L 



0 April I3» 1976 



Dear Colleague: 

To help Met the need for current infomatlon .regarding GRB 
validitystudy activltiee and to facilitate the devalifr»e«^t of «ore 
systematic and ragular procedures for assessing the predictive 
validity of the CRB* the Graduate Becord Bxaalnations Board has 
funded a three^year project designed to achieve these goals. The 
project is intended to encourage and facilitate GRE validity-study 
research in graduate achool settings iihera a variety of co«plexl- 
ties~organisatiottal» conceptual* statistical, and logistical- 
have made it difficult 'for concerned deans and faculty Btabers to 
design and conduct such studies In the past. 

Briefly, graduate schools willing and able to provide necessary 
data »ay obtain assistance from Educational Testing Service in design** 
ing validity studies. ETS will also analyse the data and report find- 
ings; institutions participating in this cooperative effort will 
receive copies of the results of the researth. Multl-instltutlonal/ 
departmental approaches to GRE validation research will be exp ored» 
e.g.. studies involving the concurrent participation of departments 
from the same set of fields A each of several cooperating graduate 
schools using a standard design and comparable data. In reporting 
about GRE validity studies, the Information supplied by graduate 
schools will not be identified with a particular institution and will 
be held confidential* * 

The information called for in tiM enclosed two-part questionnaire 
is critical for the planning and development of a cooperative effort. 
It is needed to identify institutions/departments that have conducted 
GRE validity studies in recent years and to identify those interested 
in exploring actively the possibility of participating in the coopera- 
tive effort. If you report an interest in participating, appropriate 
follow-up inquiries will be made; even if you are not interested in 
further validity worit at this time, your completion of this question- 
naire will be of great value. 

Your assistance in completing the questionnaire and in sharing 
the results of any institutional/departmental validity studies that 
have been completed since 1970 will be greatly appreciated and will , 
contribute substantially toward the goals set by the Board in funding 
this Important project. 

Sincerely yours. 



Enclosure 



^Sanford S. ElbergV' 
Chairman 



cc: Maryann A. Lear 
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GRB Mo. _^ (ITS 

QMSmXZ RBOOM). EXMtZKATIO))S BOARD. 

> 

A Survey of. GRB Validation Itosoarch Activitios md Intorosts [Part 1| 



Nuite of imtitutiw Locetioii 



4- 



KtM of retpondoatj ^ Title 

- - - r " i ■ ■ ■ » I, 

********* 

\ ^ About the Survey 

« Ihie survey is part' of the Cooperative^alidity Studies Project being conducted by 
'ETS^or the Graduate Record Eicasinations Board* 

Part 1 of the survey calls for (a) general classificatory information (e.g.» sise» 
control, nighest degree), (b) United data or estimates regarding graduate-school 
Hide application /enrollment status/GRE-score availability variables, (c) geperal in-* 
formation about the incidence and nature of GRE validity^tudy activity since 1970, 
and (d) an assessment of the general level of interest and/or concern regarding 
questions related to the validity of GRE scores, for predicting student performance* 

Part 2 of the survey calls for information that will help to identify departii^ts 
^jpr programs (a) for which GRE validation research may be relevant, (b) in %rhieh 
vaTtdity studies have been completed or are in progress, and (c) which from the per** 
spectlve of the graduate dean*s office, would be interested in exploring further the 
possibility of participating in a cooperative study. This infciiaatl* a is critical from 
the point of view of study planning^. Appropriate follow-up inquiries will be made to 
assess both interest and readiness to participate in studies* 

For your rftfen^Me, a copy of a recent GRE Program "Cumulative Suanary Statistics Re«* 
port'* prepared for your institution has been included in this mailing. This report 
indicates the total nimtber of GRE Aptitude Test score reports forwarded In a recent 
year as well as the nun^er of Advanced Test s '^re reports in- up to 19 fields. 



General Instructions 

1* Please complete both parts of the survey at your earliest 
convenience. 

2. Use the business reply envelope provided for returning the com-* 
pleted survey and any available validity study reports or 
summaries to Educational Testing Service. 

3* If you have questions about the survey, call collect, as follows: 

Kenneth M. Wilson 609-9 2 1-9000, Ext. 2391 
Educational Testing Service R208 
Princeton, NJ 08540 



Information provided In the survey will not be identified with your Institution by 
name. It will be Included In suamarles for groups of Institutions and departments* 



IN ANSWKRIKG QUESTIONS V - IX, PLEASE NtTTE T!iAT BEST ESTIMATES ARE REQUESTED. 
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1 Dbctorttt 



2 MMt«rU 



tt. 



tMtitutiofuil eofitfol or •ffllUtio* 



1 frivfttt» MCttriw 
III. number of dt|ir«M tiitrd**. 1»74-I5 *e«4«aie 





IV* Oo«t your lft»tttutto« haiw • gontral witfo« 
•iteltttona l>oltcy that •ppli— to til 
traduace drptrttttntt* t.o.. not necoMsrliy 
th# ittiwltrd but • ttmaon poUcyT 

■ Quootton V) 

If •^w," pUMt ch«<li tho stotcMtit bolW 
that U9t d«icrtb«* tho g«t«r«l tdttlMiocM 

policy of yoar tnstttutton* 

1 EMsntUlly ••opm door.* l.o.. 
- - AppUctnts ^ fce^t t*rt«tii ttlntMl 
r»quir«««ntft {t^h boldtng • 
bftch*lor*s dnr^) vro odaltt^d to 
puMut • Rr«duOt» degro* 

2 EsttntuUy "opw <>oor** tn»ofsr 

■ taklnR graduate couraaa !• concemad 
but ad«i«»lon for dagraa purpotaa la 
a a«Ucttva procaM* 

y Candtdat«i.»*atlng cartaU-ataodarda 

(e.g.. tpectftad undargraduata aver- 
age and/or CRE-tc^ra vlntmua) »ay ba 
adaltud. othara aay ba admlttad on 
an axcepttott baats, avao tf balow 
tttnta 



4 AdalMton Is on • cpwpatltlvt. cow^ 
" ptrattva ba«U vtth othar appltcanti 
•aeklng adnia^ton to a partSculsr 
ptoRtito for a gtvan ttM pattod. 



Durtut tb» Mitaalow JtW;J»^>» 
invalviiig «ppUc«iiU for Ull I9n^ ^ 
mwf applieatioM f#r atfttlaaioo ta tM 
gradiMita achoal ractivtd* ^ •••^ 
•ppliemta vara accaptad for odolaaiM* m€ 
ban My .aeeaptad applieaota octuallt 
anrolladt Coiiai4ar 4a$r«a«aro4it «»Mt* 
t90U only* 

1 Tatal a^lUMta* for foil lf» 

2 itaibor dpflUaau otfcap tod 



fl. 



S other 



»r aceof tod ofRlioooU 

" oorolllog 

Should «» individual iotarootod i« ofplyiog 
for odiAaoioa to your groduato oabool to 
purouo a dairoo prognNi» aubait OM Aptitodo 
Md/or Advooaod Toot aoorot i» ooonootioo 
uith tha oi^plieotioiit Maoaa aaloct tbo o^ 
•MNfor bolou that boot rofloeta ioo tito* 
tionol/daportoaotal proctiea (ro^tirouaott* 
•upactatiofia, ond tha Uka) with roMtd to 
Aptitoda ond Adumead Toat ocoroo* 
taapoetivoly. 

out 

Aptitttda AdvoMOd 

1 I too* oeofoo Obottld bo 

•Uboittad 

Hot acoraa oood oot 
bo atibttlttod 

AMuar dtpoodo oo oppli- 
aantU iotaodad daport* 
•aot/fiald/dagraa, 
uttdargradttota racord, ate* 



VXl. 



mrt. Quoattouo ni. Vltl. aod « coll for 
battMtiiMMOsix for eartaia gradoata 
aehooiAdda^atatiiuea for uoriablaa thtt 
tffact valtdtty-atudy plaimiog* Hia claaa 
iatarvala providad raflaet ralativaly largo 
tolorMeao for thaaa aattoataa* If ypa hptig 
aora araciat efll#d far b» tha 



g. ^;«cri;; prov 1^>d. tilaaaa chack 
tSLtot^ that ia aoptoprl *^*, 
rcfZjj pr^Mfa ^iVura in «Hca aroirldtd 



yrou t;io parapoctiira of tha groduota daan o 
off ieo, nbot io your boat aatiaata 
proportion of appltcoato. 
for dagroo-^radtt torolloant to Fall 
mttttttt CUE Aptttuda Tatt tcoraa i« cponacttoo 
with thatr appltcatton for adutMtonT 



J tatantially all 
J 90 paraant plua 
parcaiit 
4 50-74 parcant 



<or 



J) 



^5 2S^9 parcani 

6 10>*24 parcant 

7 Uaa than 10 parcaot 



O'i 
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vtn. 



of att indtvt«tu«tt «mtottifi« for tht tint tim 

your bett •ttlmata of thi» rroportloii ct«Mtft«ht0 
•s **f trtt-tlMp-«mrolM tradtuitt •tu<l#nt •**«*• i.t«t 
«#tch no pr •virus i|r«au4itt trudy •( any Inttttu- 
ttofit (0»tr*^credit ont¥, fuU- or ^rt-tlM) 




t Ett«ntl«Uy Mil (or . 

,2 90 r«rc«nt ptus 

> y5*'ll9 |>«rc«nt 

4 S(h>74 p«rc«nt 

^5 25»4f p«rc*nt 



j6 tt«t th«n 25 p«reeot 



tX. •ll fir«t-tia«-iihrolttd graduatt ttud^ntt 
enttring tn • given yMr* vhat It your httt 
•attutt of the proportion Hkaty to contlnuo 
thrtr enrottMnt into th« tacond yaar* on tlthar 
• fuit* or a part-tta# baals? 



JL Caa^nt tally alt (or, 

^2 90 pvrcanr ptua 

^1 75»ft9 p«rcant 

J* 50-74 p«rr«nt 

^5 25-49 percent 



X) 



> lata than 25 p«rcant 



!»OTE: Cueattona X. XI. XII. and XlII cail for 
.general tnfotvatton about CMT valtdtty-otudy 
{actii^^t*» and tntereat, Pttaae #nai#er the<ie 
t generM^nuest;^^s and then pr.^cteJ to Part 2 of 
the <iurv«*y vhrcb catta for nora detailed 
■ inforwatton. 



Hava any tnatttuttonat/depArtaentat atudtea, 
deat^ned tn p^rt at leant to examine the 
retattonnhtp of CM£ Aptitude and/or Advanced 
te<t acoret to any Measure of atudent 
-sutcena" In 4 decree pr^niraR In anv fteld(f>» 
been cospletr 1 at your iniitttuttun stm*» 1970? 
Are any ^•ich studies now In prr»Kr«»ss* Please 
enter one rhe<'k {¥) in^a<'h culunm* 

Validity Studlea 
Cocipleted? In Progress? 



1 



Ko (If "Mt*- tu both 

sktp to xrn) 



^2 Yes* Involving ORK 
Aptitude only 

^1 Yea* Involving ot'e or 

ai*re Advanced Testa 

onlv 



Xt* Navt any of tha atttdlt* eoa^Uttd o? U troi|rtM 
bean conaemad dlrtctly or indlrMtly with jtlia 
traltdtty of Ctt acer«a for pradtetlng gradvata 
achool auccaaa aaong mdtvtduala io any of tha 
follovtng a«*groupa? riaaga amwar fo? Meh aiib* 
group by etrcling ••yaa'* or 

Valtdttjt gtn^y in^lvitg 
tubgroup CM 9 CU 

Aptttttdat MvMcadt 

^hmti Tas^ lla| Ym^ Io^ 

ilacfc atttdonta • • • « Taa^ Ho^ Ym^ ioj 

Haslcan-Aaartcaii 

atudaota T««^|t0| , T«a^ Il0| 

F«art9-IUe«o atttdtnta • Taa^ ll*^ / Ym^ io^ 
Othar diaadiNMtagad 

groupa •••••••• Tm^Vo^ TM^ia^ 

Oldar atudanta, raantar- 
Ing tha aducattoiMt 

ayatas Ya«^ll0| Yaa^io| 

Fart*tt«a atudanta • • Taai^Mo^ Ym^Ho'i 

xn, Conaldartng the vaUdlty atudtaa that htva baan 
Initiated and/or coaiplated ainca 1970, at ^loaa 
Inltlattva ware they undartakanf tadlcata tha* 
liidlvlduat, office, atc,« priMrlly raaponalbla 
for s at ting the atudlea in ■PtIoa >' If a alogl^ 
aptlon vitt not aufflce* check each applieiibli^ 
option* 

, - ^ 1 Tha graduate dean ao4/or peraonnaX 
aaaoclated with the dean*a office 

2 An office of inatltutlonal reaaarch 

^ 3 A central adalealona of flea 

. 4 A departMAtat ehalrwia 

- , 5 A departttantat eoaAlttea 

6 A atandlng cowlttee of the graduate 

achottl 

7 A atudent coaalttae coocemod vith 
graduate achool pollclea 

S A gradtiate atudent (theala or 
diaaertatlon) 



J9 An Individual faculty 
^10 An external agency _ 

11 Other 



Yea* Involving ht*th 

Aptitude and Advanced 
Testa 



r 



4 



Si 



tt. Wh«t l» the current levef of Interest/concetn In the graduate school over 

qisMtions regarding the validity of- GRE teat a for predicting fttudent perforoance 
in <iraduate study? Please provide your as. isioent of the gi?nf*ral level of inter- 
est/concern from the point of view of <a) the dean*8 office^ <b) the graduate 
^ faculty generally^ and (c) student;/ applicant groups. 
f * 

* Level of interest/concent 

Office or Group ^ j,^^ 

Graduate dean's office 1 2 3 

* ' Graduate faculty generally 12 3 

Studi»nt/appllcant groups 12 3 

Please elaborate briefly below» indicnting the t:^pes of questions and issues that 
are involved, reasons* for concern or lack of concern* etc, • •'^ 



^ 




ORB No* (BIS w) 

of iMtitution ^ L ocatio n 



Name of rMpondent^. ^ Title /Telephooe 

Survey of GEE ValidntioQ RMMrch Activities and Interest I Pare 2 



XNVRNtOBT OF ACTIVITIBS AND INTBRBST 

XIV* Is there at least one departsent /degree progran in which at least half 0** 
entering students ordinarily have GRB Aptitude and/or Advanced Test scores* 

1 Yes (Please cosplete Que8ti<ms I, 2 and 4, inside) 

2 No (Please coaplete Question 1, Inrlde) 

Ceneral Instructions 

The inventory may be conpleted by checking or entering a code for each of severe^ 
designated fields in such a way as to indicate (a) the availability of a «aster*s«» *and/or 
doctoral-level program, (b) GRE**score availability level, following the pattern suggested 
by XIV, above, and (c) whether validity studies have been conpleted or are in progress* 
You are also asked to identify prograias ot departaents which, from,^he perspective of the 
graduate dean's office, nay be potential participants in cooperative studies* Detaiiled 
instructions are provided inside* Please note the following general instructions: 



1* In assessing GRE**score availability levels* besc estinates only are sought 



2* Please provide copies of reports of c<^Jipleted validity studies whenever 
possible* If descriptions or sunnaries rather than copies of reports are 
deemed appropriate, please use the back of this inventory* 

3* If you identify one or more departments or programs as possible partlcl** 
pants in cooperative studies, appropriate follow*up inquiries will be 
nade. No connitaents are Involved* If you are not in a position to 
specify particular departments or programs, but are Interested in explor** 
ing further questions about participation in cooperative studies, indicate 
this by checking in the appropriate space inside* 

4* If you have questions about the survey, call collect as follows; 

^ enneth M. Wilson 609-921,9000, Ext. 2391 . 

Jucatlonal Testing Service, R208 
Princeton, NJ 08540 

5* When you have completed Part 2, please return both Part 1 and Part 2 (and 
copies of reports of studies. If available) to KTS In the business reply 
envelope provided for this purpose* 



Information provided will not i>e Identified with your institution by name. It will be 
used for study planning and in summaries for groups of Institutions and departments* 

SPECIAL NOTE: IF YOU HAVE CONDUCTED OR WISH TO CONDUCT ORE VALIDITY STUDIES 
IN FIELDS NOT LISTED INSIDE, PLEASE PROVIDE RELEVANT INFORMATION ON THE BACK 
OF THIS INVENTORY. ACaIN, PLEASE NOTE TlUT ONLY BEST ESTIMATES ARE SOUGHT 
REGARDING i;RE-SCv)RE AVAIIABILITY. 



G«Mral Inventory o£ GRE Validation Ra»earch Activltlaa and Interests 

LISTED BELOW ARE 19 FIELDS FDR WHICH A GRE ADVANCED TEST IS AVAILABLE. THESE FIELD DESIGNATIONS MAY BE THOOGHT OF AS 
REFERRING TO DEPARTMENTS /HELDS/AREAS OF STUDY IN WHICH DEGREE PROGRAMS MAY BE OFFERED AS WELL AS TO PARTICULAR GRE 
ADVANCED TESTS. Answers to the questions included in this section of the survey P^^JJf * ~"P''^*?*i^* • 
overview of the status of validity-«tudy activities In the broad fields of graduate study listed and an Inventory of 
instltutional-depaxtnental areas In lAlch cooperative validity atudles •Ight be developed. 

1. Is a degree program offered in the field? Check (/) under M (Master's) In Colu« la and/or D (Doctoral) In 
Column lb, as appropriate, to Indicate at least one degree program at the designated lavel(s). 

2 Is there at least one department/program In which GRE Aptitude and/or Advanced Test scores ordinarily are available 
r f^r at L^st hSf °"! ■tll^.nt, enLr?na each year? Check under "Aptitude i Advanced" In Column 2a to indicate one 
or more departments or degree programs In which ha lf or more of enrolled students have scores on ^"'h Aptitude and 
Mv^ced T^ts. Check under "Aptitude," Column 2b, to indicate one or -ore f«P«"T"'?''°'^''T A^^i^LS" L 
stll^ents ordinarily have Aptitude scores but not Advanced Test scores. Check in Column 2c, ""«»«^ JJ^JJ^J^ 
indicate availability of Advanced but not Aptitude scores for a majority of students In one or more departmenta or 
programs. 

SPECIAL note: ENCIRaE A CHECK MARK IF THE LEVEL OF SCOHE-AVAILABILITY TOR ANY DEPARTMENT /PROGRAM APPROACHES 
ESSENTIALLY COMPLETE COVERAGE (e.g., due to patterns of requirements). j 

\ For anv proRram checked in 2a, 2b, and/or 2c, have institutional/departmental validity studies [to assess ^he 
SeL^o? vSuJity Sf^RE Tea (s) for predicting student "success"! been completed -ince 1970? Are any validity 
stSSHs in Xxoitl^t USE "C" TO DENOTE A COMPLETED STUDY and/or "P" TO DENOTE A STUDY IN PROGRESS. Report In 

3a tto^e nidies that involved both the Aptitude and an Advanced Test. In Column 'b. report studies 
involving the Aptitude Test only. Studies involving only an Advanced Test should be reported Jol^^J^; 
Studies involving the GRE Aptitude Test in sample, that are not homogeneous with regard '° "•"(^*!5"?J' J*-'- 
students from several social science departments) should be reported In spaces provided under Column 3b In the 
last three rows of the fornix below* 

SPECIAL note: FOR CO J'L TED STUDIES PLEASE PROVIDE A COPY OF EACH REPORT OR A BRIEF SUMMARY OF STUDY DESIGN AND 
FINDimS. . >R STUDIES IN PROORESS PLEASE PROVIDE A BRIEF DESCRIWION. Encircle P aml/^ 
»nfr4»« M yo.. ar.. including a vallditv-stud v report, summary, or description with your survey foym . 

4. A GRE validity study may be relevant for a department or program if Jrr^JeiaJ'tlniJitu- 
Columns 2a. 2b, and/or 2c." From the .perspective of the graduate dean's office, in which °^ 

aoS/deSirtm^ntal areas i« there currently active interest in the validation or ^""her validation of GRE tests 

"fedicKJi " Jtudont performance? Designation of an area as actively Interea ^ involves no co^^ 
course but should reflect the dean's Judfjment of institutional-departmental readiness to explore "^"^"^^ 
poSriMuS Tf panic P^Ung in co'^perati^e GRE validity studies GIVEN mutually study models and pro- 

^^.J"!' InIoE^ A 'Silt? ;S ™e'nAME(S) of all ACTIVELY JJJJ^^^TJJ^JSf ^^al'tS^^ 
departmental/proRrammatic emphasis is on admission/selection/r.RE ^""^ ""'"^J "JJ^n^ 

study (D), or both (M i D), by adding the appropriate letter(s) after °f^'^%S";*"''" 
If interested in possibility of a validity study, but unable to name specific areas, check space provided in 



M 



ERIC 



For dttailtd instructloiiSt pleaae t%Ut to QutstioM 1» 2» 3» and 4» ra0pactlv«lr» abova. 



Flald/ 

departaant/ 

araa 


Degrae 
prograiii(a)? 


GRE acotas avallabla, 
any dapt« /program? 


Validity study since 
1970? 


Inatltutlonal/dapartaantal araaa intaraatad in vali«* 
datlon or further validation of GHB Aptituda and/or 


n 


11 


Apt* 

Adv, 


Apt« 


Adv« 


Apt* 
Adv. 


Apt. 


AdVs 


Advancad Taata? Sntar nam^a) of daoartnanta/ 
programa and Mt D» or M & D aa appropriate • [Saa 
alao category 4a) at bottom of form]* 




(lai 


(lb> 


(2a) 


(2b) 


(2c) 


(3a) 


(3b) 


(3c) 


(4) 


24 BIOLOCY 




















27 CHEMISTRY 




















31 ECOKOHICS 




















34 EDUCATION 




i 
















37 ENGINEER INC 




— — -t 

1 
















44 FRENCH 








1 


i — i 










52 GERMAN 










* 

1 i 








* 


91 SPANISH 




I 






i 

. i 


J 


1 






46 GEOGRAPHY j 


















47 GEOI^Y ! 


I 


1 










52 HISTORY : ! 
















$7 LITERATURE • j j 

— enR\^f}h ; .1 
















67 MATHEMATICS : j j 






I 










71 MUSIC i : 1 






i 
I 










74 PHILOSOPHY 


T 
1 

t- — 

1 

t 


1 
















7? PHYSICS 






i 


l~ J 




■- f 

! 




79 POLIT SCIENCE, 








— 1 






1 

! 




81 PSYCHOLOGY j 


















67 SOCIOLOGY 














t 





COMBINED FIELDS: TO BE USED FOR REPORTING 
RECENT VALIDITY STUDIES IN 3b» GRE Aptitude 



4a) t 



1 Check here if Intereated in 



11 Sample from two or more natural scienca^.f ields 



12 Sample froa two or more social science fields 



13 Sample from two or more humanities fields 



XV. TO WHOM SHOIJLD FOLLOW-W INQUIRIES ABOUT VALIDITY STUDIMS ADDRESSED? 



exploring possibility of participating in 
a cooperative study t'liut not in a petition 
to identify particular departments oi 
programs at this cime. 



_l Respondent named on cover page 
2 



Name/T i t le/Te lephone 



ERIC 



^3- 



2«4 



Please use this |>age for descriptions of validity studies » for elaborating answers 
to questions » or for Identifying validity study areas not covered by the Inventory^ 



Appendix B-2 

SURVEY OP GRADUATE SCHOOL VALIDITY STUDY ACTIVITIES AND INTERESTS 
. A SUMMARY OF- FINDINGS 

Infonuelon o««dtd for project plenniag and developoMnt 
wie obtained through a survey of graduate deans of institutions 
comprising the nembership of the. Couneil of Graduate Schools 
(CGS). the survey «fas designed prlrarilyt (a) to identify prospec- 
tive participants in cooperative validity studies, (b) to ascertain 
the types of questions cr issues a^out GRE validity that were 
current oncaapus, and; (c) to obtain Information regarding current 
and recent levels of validity study activity, especially since 
1970* 

With a c'overing letter from the Chairman of the Graduate 
Record Examinations Boards the survey was mailed in April, 1976 
to deans of 344 CGS member schools** A total of 244 deans (or 
their representatives) responded. Some degree of interest in the 
possibility of participating in cooperative studies'^ was indicated 
by 130 of the respondents. The role of the survey in identifying 
prospective participants in GRE validity studies is considered in 
a subsequent section* However, attention is directed first to 
information provided by the survey regarding the status of OEtE > 
validation research in CGS member schools* 



Status of GRE Validation Research 

The fact *>hat a large number of schools (i*e*, 130) indicated 
some degree of lnter«»8t in the possibility of participating in 
cooperative GRE validity, studies may be understood best when 
considered in relation to the extremely low Incidence and uneven 
nature of local » institutional/departmental validity study activity 
reported by survey respondents* The survey Included questions 
regarding (a) the extent of validity study activity since 1970, (b) 
studies that may have been completed for subgroups defined In terms 
cf variables such as sex and/or ethnic group membership, and (c) the 
individuals or offices responsible for initiating and conducting the 
studies that had been made or were underway* In addition, it was 
requested that materials descriptive cf completed or current studies 
be forwarded. 

Judging frobt the responses to these questions, summarized 
In Table I, and the nature of the descriptions and reports forwarded. 



Insert Table 1 about here 



*A copy of the covering letter and the survey forms used are 
Included In Appendix B-1 (q.v.)» 



Tablt 1 

Data on GRE Validity Study Activity Sine* 1970 In 244 CGS^Mtabar Schools 



H«v« «fir lnscltu(lofluil>Utt^art«Mfn(iil ^(udl^s» 
deslgntd In part at Um( (o examine t\%Q 
r«la(lonshlp af CWi Ap(l(u4c «nU/or Advanced 
Test tcortts to .tnv mamiro of'studirnt 
**iiucc«ti»** In .1 dcftrcM progrdm In 4iny fl«ld(s)t 
hi»en comfiUt«a m your instlcuclon nlncd U;o? 
r« 4ny uucli NtuUl«*% nov In progrvgt? Fl^a^* 
er one oln«ck .(t) tn e«ch colu«m. 

o. reporting 

/•lldlty $tudi4»s 
CMplacodt In Progrets? 



167 




10 



1421 



177 



17 



10 



No 



Ym» Involvlnn ORE 
Aptitude only 

Yoo» involvlnii one or 
more AdvAneed teets 
only 

Yeii» titvolviuH both 

Apt t tilde and Advanced 



j[37) No answer 



Have any of the ntudlen completed or in prottrcjie 
been concerned directly or indlr«*ctly wttli die 
validity of mU; acares for predicting HrtiliLtte 
achoiil siiccene among ludlvidu.iU In tiny of 
folhiwlng Hubgrmtpiit Tleaae answer far eat ti nub* 
group by clreltug ••yea*' or **no.** 
r 

Valtdttv atudy invntvlng 
SiUtgroup UKK <.KK 

AptitiHt«*? ' Aiiv«iiH*ed* 

Uomen •••«'•••»• Yen 9 V%*^ 4 

Hack at«ulenta • • » • Yea 7 Yea 1 

MexlCdn«*American 

atudenta Yea ^ Yen ^ 

fuarto-iUean atudenta • Yea ^ Yva ^ 
Other dU Advantaged 

grottpa Yea 2 Y«»h X 

OIdi*r Htitdetica, reenter* 
itxfi the ettucatlonal 

ayatcm Yea 4 Yea 4 

Farftiflu atudenta • • Yea 4 Yea 2 



ConalUvrlng the validity atudlea that have W«n 
initiated and/or conpleted aince I970» at who^e 
initiative wt*re they undertaken? Indicate cht* 
indivtcluat, (>ffice, etc., prinuirily reaponniMe 
for ^i!ttii it* the Htudio'i tn tm^ri.u i^ Tf a HU»»-le 
optioirwill net aiitfice^ check «».u*h appltcuhle 
option* 

Tiie »*raduate dean »indAir pcraonnel 
UHHOClated with the de tn's nffit-o 



An office of institutional teaeurcl* 
A central admiaalona office 
A de:>artmantal chairman 
A departmental commiit\'e 

A a.inaing cooiftittce ot the graduate 

school 

A ^tutJent commictoo . n%-%Tn«d wtrh 
.;rAuuatis acUool policies 

\ 'ii-ti^^tc Htiident (t;uHts or 
vlinHiirtation) 

Ai| inJ-ividual faculcy mcPtbcr 

An oxternai agency . 




2 
3 



other 



8 Dean/departmenual committee 
2 Dean/chalnaan 



1 Dean/Office of Institutional research 
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only « £«ir |r«duAt« tehoolt hav* conducted syttMUtic studies 

of tho predictive validity of 6RE scores' (and possibly of other 
preadnissions variables, such as undergraduate 6PA) in any prograa 
during the period 1970-76. 

Specifically, the survey asked, ^Have any institutional/ 
departnental studies, designed in part, at least, to exanine the 
relationship of 6RE Aptitude and/or advanced Test <tcores to any 
8.easu?t of student 'success' in a degree prograa in any field(s), 
been completed at your institution since 1970? Are any such 
•tudi«is now in progress?** 

0 Only 38 respondents indicated that studies involving either 
the 6RE Aptitude only (18 schools) or both the Aptitude 
' and Aidvanced Test (10) had been completed; studies in progress 
wsre reported by 30 schools, including some of the schools 
that reported completed studles<^17 invol^d the Aptitude 
Test only, 3 an Advanced Test only, and 10 involved both 
the Aptitude and' an Advanced Test. 

The question vhich elicited this response framed a very 
"inclusive" definition of "validity study." It was intended to 
permit an affirmative response if any empirical examination of 
variation in student performance by GRE score-levels, or vice 
V versa, had been undertaken, and documented sufficiently to warrant 
circulation Intra-institutlenally (e.g., as a memorandum, report, or 
tabular summary) • 

Studies not comprehensive . Both the low incidence of reported 
validity study activity and the* uneven nature of the xhibits 
forwarded as descriptive of that activity reflect the essentially 
undeveloped state of the "validity study art" In graduate school 
sectlngn. 

o Only 10 survey respondents Included materials descriptive 
of completed, curx'nnt, or planned local studies of GRE 
predictive validity. Of the ten exhibits forwarded, only 
on»* involved both a systematic analysis of relationships 
mong cleazly defined criterion and predictor variables, 
and samples broadly representative of the respective functional 
divisions of the graduate school. 

® The exhibits differed markedly in format, comprehensiveness 
of reporting, and dassif lability as ''validity studies." 
Materials forwarded as lllustratlv* of' local GRE valldl.ty 
study Activity Included, for example, two summaries of 
descriptive statistics on grades and GRE scores, by depart- 
ment, that did not consider relationships among the data 
elements described. Also Included was a Xerox copy of a 
computer printout of a table of intercorrelatlons for one 



ERJ.C 



4 



as-. . 



Mapl«; a tcacttrploc of ORB tcoCfs «ad CPA and « brltf 
MBoraodua eoBMaeiiig on aa obt«rr«4 eorr«l«tlea in ont sMiplt 
In on« fltld; «xc«rpc« from « gradu«c« ttudant't aMChods«-proj«cc 
thMls raporc; two ad hoc acudlaa In achooU of Bducaclon; 
raprinca of pubXiahad acudy caporta baaad on acudanca ancarlag 
during cha 1960a. > 



Subgroup atudiaa not availabla. In vtatf of cha occaalonal, 
ad hoc nacura and llnitad acopa of cha **valldicy acudy procaaa** 
*ganaraXiy» aa Infarrabla froa cha foragoing anaXyala, ic la anclraly 
undaracaodabla chac only a faw raapondanca raportad acclvicy daalgntSd 
CO anad Ughc- on aora conplax quaaciona thac arlaa ragardlng cha 
(somparaclva pradlcclva valldicy of GBiZ acoraa (or othar adaiaalona 
varlablaa) for groupa daflnad in carma of aax» dlaadvancagad a.cacua, 
aga and/or dagraa of concinulcy of graduaca acudy* anrol Inane acacua 
(full-varaua-part* cliaa), acc* 

Aa Indlcacad in labia 1, only a handful of achoola raporcad 
cnac any of cha acudlaa undarcakan alnca 1970 had baan 
concamad diraccly or Indlraccly wlch quaaclona ragardlng . 
cha pradiccive valldicy of ORE 4pclcuda acoraa for woman 
(9 achoola), black (7), HaxlcanrAoarlcan (3)» Puarco Rician 
(3), '*ochar dl8advancagad"(2), oldar (4), or*parc-cina (4), 
acudanca* raapacclvaly* Evan favar achoola raporcad axsalnacion 
of cha valldicy of COlZ Advanced Taaca for auch aubgroupa. 
tlona of Che axhiblca forwarded involved analyaaa by aubgroup* 



itaaponaibilicy for Scudiea 

It l0 reasonable to infer from Che foregoing that a ^Walldlcy 
study function** co;:ninues to be an undeveloped area in graduate 
scnools generall]r*-»l#e*» validity studies are not conducted regularly 
as part of a process haying clearly perceived organizational, 
conceptual • and operational parameters* Other survey findings 
support this inference* For example, the few studies that have been 
undertaken reportedly were initiated by a variety of different 
individuals and offices (Table 1): 

^ The graduate dean and/or personnel associated with the 

dean^s office were designated as primarily responsible for 22 
of the studies completed or underway* 

x. 

13 studies were initiated by departmental commlctees> i by 
departmental chairman^ 3 by individual faculty members^ and 
2 Dy graduate students as projects associated with the programs 
of study* 



^ An offict of lafltitutioiMl r«tMtch or e«sel&g (•vmlu«tioa 
tMAtcxi) And « conttU adaltslbiit' offict mm eltod on* 
* tiiM each as Involvad in currane or eoaplaead atudlM* 

Xha ^graduate daan's off lea was eha siagla noat fraquaatl^r 
cited initiator of 6RE validity studies* bat represenccn Ivas of 
schools or departments within the graduate school were reportedly 
responsible in almost as maiqr cases* The complex, decentralised 
nature of the graduate school clearly has militaeed against the 
development and implementation of a graduate*school wide approach 
to examination of the validity of admissions decisions based on 
GR£ scores or other evidence of the qualifications of candidates 
for admission to graduate study. At tha same time* it would 
appear that tne graduate dean's office tends to have a higher 
degree of interest in quastions bearing on the validity of ORE ^ 
te«t8 (andi other data) for predicting student 'perfocmance than 
repr^entatlves of the respective departments* 

Current v^uestions and Issues Regarding GR£ Predictive Validity: 
Deans' Assessments 

♦ 

The survey sought inforaation regarding some of the specific 
questions and issues pertaining to 6RE predictive validity that 
• are currently of interest and concern to graduate schools as 
viewed from the perspective of the graduati* dean* In addition, » 
deans were asked to assess the general level of ..Interest and concern 
regarding these questions (a) in the dean's office, (b) on the 
part of departmental faculty, and (c) it student/applicant groups* 

As Indicated in Table 2, only 33 deans reported a "low" 



Insert Table 2 about here f 



vel of lat'erest while 94 reported a "high" level of .interest 
GRii validity-related questions; tney perceived a somewhat lower 
el of Interest! in sucn questions among graduate faculty generally 
• in student/applicant groups* 

About 150 of the respondents provided some elaborative commentary 
in connection with their assessments ot the general levels of 
interest and concern regarding GRE predictive validity* .-lany of the 
comments were relatively general in nature, referring to local 
patterns of GAii use .father than Co validity-related concerns* For 
example: « 



Table 2 

Question Regarding Dean*8 Perception of Level of Concern Over 
GRE Validity* and Associated Distribution of Responses 



Wliat Is the current level of Interest/concern in the graduate school over questions 
regarding the validity of GRE tests for predicting student performance in graduate 
study? Please provide your assessment of the general level of interest/concern from 
the poi«t of view of (a) the dean*s office, (b) the graduate faculty generally, and 
(c) student /applicant groups. 



Not 
responding 



Level of interest /concern 



13 ( 5.3%) 
13 ( 5.3%) 
38 (15.6%) 



Office or Group 
Graduate dean's office 
Graduate faculty generally 
Student/<npplicant groups 



33 (13.5%) 
60 (24.6%) 
95 (38.9%) 



Low 



lOA (42.6%) 
131 (53.75t) 
78 (32.0%) 



Medium 



High 
9U (38.5%) 
40 (16. A%) 
33 (13.5%) 



V 



**SoflM departments place mora ^phafls on the Graduate 
Record Bxamtnattona than otherftome have a cutoff aeore and 
will not accept applicants «ho do not meet this requirement*** 

**Most scores are used here as a basis for admission 
since there Is a heavy demand for places*'* 

"Although 6RE scores are required from all degree program 
applicants, other factors are given equal weight In predicting 
chances for success In the Gradoate School*** 

Reference to **oplnlon,** "belief ►'* or **convlctlon" regarding 
the '*usefulness,'* •'value,'* "validity," etc*, of GRE scores was a 
frequently recurring element in other general comments emanating 
from graduate Ichool settings In which no validity-study activity 
was reported* 



"Since the Graduate Faculty Is unwilling to impose 
a unlverslty-wlde requirement for the GRE,' the Interest 
In validity studies Is limited* Departments who use It \ 
think It Is valid; those who don't, think little about It**^" 

"Hany faculty members do not feel the GREs reflect 
the students' predictable performances*" 

"Since we are not bound by automatic cut-off scores, 
the concern of the faculty for the validity of the GRE is not 
particularly high* They have in their own minds determined 
what it is worth, although their opinions vary* 

"Generally, the graduate dean believes verbal aptitude 
scores are very reflective of potential ability of master's 
students*** Some areas (e*g*. Psychology) agree, but many 
faculty do not and believe they ought to be eliminated*" 

"Most everybody, if asked, will express reservations about 
the usefulness of GRE scores* However, the level of concern 
docs not extend to the making of unsolicited' proposals for 
(validity study)*" 



Reliance on subjective evaluations of predictiv e validity 
13 ImBllclt in such responses * 

A number of respondents cited particular foci of concern. 
Interest, or controversy In connection with the use of GRE scores 
in admissions including the following: 




1) Ax% tests valid for prsdietiag ths psrfoiaanct of 
tthaie olabrleiss or dissdmtagttd seiadottj^? Ol^sr 
or p«rt*tlM students? Forslgn studonts? 

• t 

2) Should uniform cutoff scores be used? Should weighted 
combinetions of Verbal and Quantitative scores be employed? 
In general, what do 6RE scores "mean'* and what is the 
best way to consider scores in the admissions process? 

Difficuitiss in conducting validity studies were mentioned 
by a nufliber of respondents who cited such factors as criterion 
inadeouacv (e.g., '*no variation in graduate grades'*), data eollee« 
tion or ob leas (e.g., "no computerised student personal data summarisa- 
tion possible at present," ''lack of clerical assistance"), and small 
samples. 

e 

Other respondents suggested that as standardised measuras, 
GRE scores should* serv^ aa objective markers of student ability- 
levels (a) to help compensate for variations in the gradihg standarda 
of undergraduate institutions, (b) to help monitor standards in the 
face of "grade Inflation" . at both the graduate and undergraduate 
levels, and/or (c) to help maintain and/or monitor "standards" among 
sevetal disparate departments. 

Complex problems are faced by graduate schools Interested 
In developing systematic approaches to evaluation of the validity 
of. admissions decisions generally, or In connection with the 
specification and maintenance of "standards." These problems are 
Summarised rather succinctly in the conments of one dean:. 

"Our concern is that we develop a balanced and far- 
ranging set of criteria for evaluating a %7idely disparate 
spread of applicants for widely dlspiirate programs ranging 
from Anatomy and Anthropology to Theater Arts and Urban 
Planning . 

We undertook to require the GRE, Aptitude as a uniform 
requirement for adoissioh... (a) to give additloual Information 
on Che iacreaslng number of applicants from P/F, MR, Honors 
Exams schools and the like and, (b) to give us some counterweight 
to' 'inflated grades'— or at least some additional standard of 
calibration. 

In Art (sculpture, ceramics, painting), the ORE Aptitude 
may have little application; in Art History, it may have 
high correlation. In Dance it may prove to have Ixttle use; 
in Economics it may have an important impact. And so I 
could go on throughout our 75 graduate degree-granting programc." 
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Mot all tv«dttAt« ■eheol% prM«at mu.h « wtdMMSliit ••t of 
prograas, but tho eoBploxltlos r«floet«d in t)i« forogolng ceanont 
m to soBo dogroo ch«r«et«rlstle of aott gr«dtt«tt Mttlngs* 

It 'iiottl4 appoar from naay of tha eoateata that in tha graduata 
aehool cosaunlty oplnlona and bbllafa about GBE **valldlty" or '*lack 
of validity" ara atrongly hold daaplta tha fact that, atudlaa dealgnad 
to aaaaaa pradlctlva validity In raprasantatlva **ua« eontaxts** ^ava 
not baan ^da. Pradlctlva validity fraquantly appaara to hava baan 
parcalvad, arronaoualy, aa an abaoluta taat-quallty rathar than an 
axpraaaion of dagraa of ralationahlp batwaan two or vore falllbla 
•aaauraa (a pradlctor auch aa tha 6RE and a crltarlon auch aa tha 
Graduata Grada Point Avaraga), In particular aaaplas* Ganarally 
spaaking, both tha comnanta and tha flndinga ragardlng GRE validity 
study activity auggaat that.quaationa about <GRE pradlctlva validity 
ara not parcalvad aa racurrlng quaationa to wfalclv currant anawara 
fraquantly will ba naadad. In thaaa drcuaatancat, tfta goal of 
obtaining up-to-data anplrlcal avldanca ragardlng GRE-Valldity la a 
challanglng ona* 



/ 
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Appeadlx B-3 

Siatement Regarding Confidentiality 
9nd« Study Procedures 



* COOFEHATIVC GEE VALXOITT STUDIES Btgin B-3 

ONE-YEAR MODEL i of 10 ^SM 

EduemtloMl tMClag Strvict 
P)rliietton» HJ 08540 

r 

To: PAKTICIFANTS IN GRE VALtOXTT STUDIES >l 

1 ' ^ • 

Subjtcc: Trtatnnt of 'data on IndividiMiU for purposes of vmlidation rosoarch, 

!• CRE valldacton rtsoarch roquiros tho llnluig«^ of inforvatlon about tha 
scoras of individuals oa ORE Aptituda and/or' Advaaead Tasts and othar 
pradietiva Masuras**with inforsatioa about thair parfonunca in graduata 

adthobl. 

Provisioa of auch iaforaation to organiaatioaa such as ETS is parmiasibla 
uadar axlatiag Federal legislation for purposes of **developing,. validating 
or adttialateriag predictiva testa tend for certain other daaigoeted purposes ^ 
if mch studies are conducted in a manner as will not perait. the parsoaal 
identification of students and their parents by persons oth*t tb* ^ 
reprasanrativas of such organisations and such infonaatioo viU be destroyed 
when no longer needed for its original purposes].** 

2. Farticipanta in Cooparativa Validity Studiaa are asked to ^lubnit information 
about tha scores and performance records of students on validity study rostara. 
It should be noted that the na^a of students are not req^^red to carry out 
the validity study analyses* and institutions may elect to eliminate namaa 

/ of studerta from the copies of rosters submitted to ETS. Some type of 
identification that will permit resolution of possible questions regardirg 
missing* out<^f«range» or improperly cdded data ahould be aubatitutad in 
such casaa. ) « , * 

3. ETS procedures will be designed to protect the confidentiality of individual 
data in all cases. For iastittutiona that elect, for any reaaout ro submit 
rosters coi^taining namaa of atudentSt the following procedures will be 

« followed: 

a) After initial screening by prdject staff, for monitoring and 
editing purposes, data will be prepared for machine processing 
with numeric identification substituted tor name identification. 

N 

b) Original data rosters will be retained in a secure place for 
reference as required to resolve data-related queations that 
may arise during the course of the validity study process • 

c) Original d^ta rosters will be retained under secure conditions 

no longer than Is requited to complete the sequence of activities 
involved in the vatidation research project and following 
completion of such. activities the original rosters will be 
destroyed* ^ 

4. Nases of Individual studenta will in no way.be Involved in reports of validity 
study findings* 

5. Names of institutions will not be identified with specific validity study 
^. fj^ndings in summary reports prepared for general distribution* 

Sovembar, 1976 
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Statement Regarding Confidentiality 
and Study Procedures 



COOPEKATIVC ORE VALIDITY STUDIES B«gin B*-.^ 

liN^>iL^;AJl Mi>DK^^^ p^g^ 1 10 pages 

SducatlotuI Ttsdng $«rvlcft 
Fflnctton, KJ 085**0 



To: PAKT::IPANTS in CRE VALIDITY STUDIES 

Subject; Tfeattsenc of data on Indlvlduala for purpoeae of validation raaaarch 



1. ORE validation raaaarch requires the linkage of inforaation about the 
scores of iijdividuals on ORE Aptitude and/or Advanced Tests and other 
predictive oieasures with InfonMitXon about their perfonaance in graduate 

school. 

Provision of such infor«atlon to organizations such as ETS is permissible 
under existing federal legislation for purposes of "developing^ validating 
or administering predictive tests [and for certain other designated purposes^ 
if such studies are conducted in a manner as will not permit thd petsooal 
identification of students and their parents by persons other than 
representatives of such organizations and such information will be destroyed 
vhen no longer needed for its original purposes]*" 

2* Participants in Cooperative Validity- Studies are asked to stibmit information 
about the scores and performance records of students on validity study rosters* 
It should be noted th4t the names of students are not required to carr/ out 
the validity study analyses* and institutions may elect to eliminate names 
of studentMfrom the copies of rosters submitted to ETS* Some type of 
lientirlcation that will permit resolution of possible questions regarding 
missing* out«»of-range, or improperly ctded data should be rubstituted in 
such cases. 

3. HTS procedures will be designed to protect the confidentiality of individual 
data In all cases. For institutions that elect* ior -r.y reason, to submit 
rosters containing names of students, the following procedures will be 
followed: 

a) After Initial* screening by project staff, for aK^uitoring and 
<*diting purposes, data will be prepared for t&achlne processing 
with numeric identification substituted for name identification* 

b) Original data rosters will be retained in a secure place for 
reference as required to resolve data-related questions that 
may arise during the course of the validity study process. 

c) Original data rosters will be retained under se<^ute conditions 

no lcn;?er than Is required to complete the sequence of icclvlties 
Involved In Che validation research project and following 
ccmpleclon of such activities the original rosters will be 
.iestrcyed. 

NUres of Individual students will in no wav be Involved in reports of validity 
study findings. / 

5. Vaaes oi ln*5t Itut ions will not be lientified with specific validity study 
v'lndinRS in suriaarv reports prepare*! for *;i»neral distribution. ^ 

November, 1976 
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^ lASlC COOFOtATIVE CItC VALIDIH STUDIES 

Ovtrvlw of Validity Study tau 
K«^ulriMQC« and Froctdurtt 

A dtulltd ttt o( Ifiatruetlou for participants in tht batlc Cltt Validity 
Studiti Projact la attachad. Tba purpoaa of thli ovarvlav la to provida a 
briaf daacrtptlon of raqulrasaata and procaduraa In ordar to pamlt an aaaaaa* 
••ttt of tha typaa of data r^ulrManta and optlona involvad for partldpanta 
10 atudlaa* Inatltutlona/dapartaanta ara axpactad to provida dau accordlot 
to procaduraa outUnad* ITS will analyta data and prapara a raport without 
cost to partUlpanta* ^ 

tha following appllaa to aach participating dapartaantt 

I. Focus of tha atudy la on flrat^tlaa graduata studanta anrollad In a 

dagraa program, and claaslf Ubla aa full«-tlna according to InatltutlonaU 
daparcmantal daflnltlona at tlna of antry Into tha dapartaacft* 

IX. Tha saapla to ba studlad consists of all such studanta who aniarad In 
fall «74 and 1975. At laaat 25 of thaaa studanta should hava CRB 
Aptltuda and/or Advanead Tast scoraa and at laaat ooa naaaura of 
parfonsanca In tha dapartaant. 



III. Tha validation parlod Is to bm tha first yaar of study. For aach 

studant antaH4M in Fall 1974, information ragardlng prograaa in tha 
dapartaant ai of Fall 1975 la to ba provldad; for thosa antarlng In 
Fall 1975, prograaa la to ba ancodad aa of Fall 1976-77, 

IV, A prograaa coda Is to ba racordad for aach atudant In tha aanpla and at 
laase ona maaaura of parforaanca ahould ba racordad; aavaral optlona 
ara provldad* 

Thus» miftioun raqulraaanta fjr pilrtlclpatlon In tha basic validity atudlaa ara 

4s f;>llows: 

A. List all flrst-tlma anrollad, full-tlaa^, dagraa-saaklng atudanta 
antarlng in 1974 and 1975. ' 

B. Encoda prograaa as of tha baglnnlng of tha sacond yaar following 
Admission for aach studant llatad. 

C. Racord GRB Aptltuda and/or Advanead Tast scoraa for aach studant 
as avallabla (at laaat 25 racomaandad). 

0. Racord at laaac ona maasura of parforaanca for aach studant: a.g.^ 

1. Firsfyaar graduata grada point avaraga or soma function tharaof 

Farformanca in soma critical coursa, coursa saquanca* samlnar, 
or common first**year projact 

3. Farformanca according to ragular and-of-yaar dapartmantal 
rating or axamination procaduraa 

Ad hoc ratings by faculty mambars according to" ona of two 
standard schadulas or to soma otha* schadula davised by a 
dapartaant « 

THE FOREGOIHG REPRESENT CORE REQITRIMENTS. PLEASE SEE THE DETAILED OITLINE FOR 
SC(X;ES:ED or RFCOMKENDEO codes, rating PROCEDlTtES* ETC, 

Ravlaad 10/76 
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BASIC COOPERATIVE CUE VAlIOITt STUDIES / 
^ Xn«tructlon4 for CoapUtlnf Validity Study Rotttrti 0««^ftltloM and FroctdutM 

l« Dtflnltlon of tormo' 

A. Flrtt^tlwo onrollod graduato studoiit: Ko tradutto atudy ptlor to 
anrollmant in dapartMnt/program* Som latltuda dUerotlonary with 
dapartMnt In Including Individuals with llmltad pravloua graduata 
. vork If such Individuals pursua flrat^yaar tasks sUlUr to tboso of 
flrst*tlM Snrollad studonts* 

B- Full-tl»s: Studtnts elasslflabla as ••fuXl-tlas'' graduata atudaats 
according to Instltutlonal/dapartMntal erltorla« 

C. Dtgrat-saaklng: Taking work crodltabla toward a graduata dagrsa and 
consldared by the Instltutlon/dopartaant to ba prospsctlva dagraa 
candldatas. If a dapartaantal sa«pla Includas both prospsctlva 
mastar's- and doctoral^dagraa candldataa, and If flrat^yaar tas)^ 
and/or avaluatlon proceduraa ara not conparmbla (or thaaa two groupa* 
tha degraa objactlva' of a atudant ahouXd ba codad aa an optional 
data alamant Caaa Instruct lona rala«:lng to Roata^ Col4ma 16 and 17» 
balow)* 

IX. Proctdures f6r computing validity study rostars: ona for aach partielpatlng « 
dapartmant* tnstructlona for aach Rostor Coluan ara as follows: 

* f 
ROSTER COLUMN 1. Idantlf Icatlon ; List all Fall 1974 and r^ll W7$ antranWt raspa^tlvaly> 
elasslflabla as flrst«tl«a anrollad* full-ttos. dagraa<-aaaking ^ 
studants at tlms of antry. t Kana Idantlflcatlon n ot ragulrad for - 
validity study purposaSt per sa . Saa stattatant re traatnant of data 
on individuals >1 tf studants for whom English Is not tha natlva 
languaga are Included they should be Identified by coding as an 
optional data element (see instructions for Roster Columns 16 and 17» 
below). 

ROSTER COLVMN 2. (Optional) Code for sex * 

Female • 1 
Kale - 2 



ROSTER COLl^ 3. (Optional) Ethnic group code 

I. American Indian^ EskimOt or Aleut 

* 2. Black or Afro-American or Negro 

3. Mexlcan-Amerlcan or Chlcano 

4. oriental or Aslan^Amarican 

5. Puerto Rlcan 

6. other Hispanic or Latin American 
7« White or Caucasian 

8. Other 
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ROSTER COIUMN 4. 



i£2iHS.i^« Mch studMe lUt*d encodt Infonatioii Mgarding 
•tatus an qf (ha baginniog of eha aacond yaar following admission*- 
i»a,» for 1974 aneranea* encoda statua aa of Fall 1975; for 1975 
antranta, ancoda atatua aa of Fall 1974, aa follovai 

4 • Continuing in prograaa covard a dagraa. in* tha dapartaant 
J • Hat continuing in tha dapartMnt; cosplatad flrat yaar in 
good atanding 

2 • Hot continuing in tha daiNirtMnt; coaplatad first yaar vith 

ona or oora inCicationa of marginal or slibatandard parfotmanca 
I - Hot continuing iit tha dapartmant; did not complata firat yaar 



KOSTtR COLUMHS (5)» (4), and (7) ara for racording GRE Aptituda and/or 
i^yrH^ ^••^ tcotaa, aa availabla, for aach atudant liatad, RECORD 
SCALED SCORES OHLT (a,g,. 520, 780, ate); Indicata nama of Advancad 
fiald on tha cover ahaat (aaa instructiona on ahaat). * 



ROSTER COLUMH 5. Entar ORE Varbal acalad acora in Coluan 5, 

ROSTER COLUMH 4» Entar GRE Quantitative acalad acora in Column 4, 

ROSTER COLUMH 7* Entar GRE Advancad Tj»at scaled score in Column 7, Identify 
Advanced Teat field on cover aheet* 



RECORD ONE OR MORE CRITERIOH SCORES FOR EACH STUDEHT LISTED. Roater 
coli^ (8) through (14) are provided for recording one or more criterion 
scorea. aa available, for each student, Scorwa ahould reflect the aaaign* 
mtnt of a stud^t to one of two or more ordered groups or categoriea in 
tarns of leyel of performance (success, attainment, achievement) during 
DUCT rjrUD?*^ ^ CRITERIOH MEASURE IS HEEDED TO COH- 



ROSTER COLUMH 9, 



Several criterion, maaaurea are auggaated, aa follows: 

ROSTER COU'MH B. Overall Gr^^t^atf Grades. (CPA, general) Performance aa reflected 
in graduate grades, baaed on work completed during the firat year 
(a grade point average or some function of gradHs earned such as, 
for example, Vrcent of gradea that were Af or A"; ••all grades 
satisfactory • 1 versus one or more grades unsatisfactory or 
marginal - O,** etc), DESCRIBE SCAIES ARD CODIHC PROCEDURES 
OH THE COVER SHEET. 

Grades in critical area. (Critical CPA) Performance in a critical 
course, course sequence, seminar, or project required of all or 
moat first-year students, or normally completed by such students. 
Grade received in such a critical area. Pass • 1 versus Fall • 0, 
^L^i^!)**^ Indication of standing should be reported in Column 9. 
^™^!LII!1^ CRITICAL AREA, CODIHC, AMD RELATED PROCEDURES OH THE 
^COVER SHEET. 

ROSTER couwM 10. Regular faculty r.Mnff, (Regular departn.ne.1 .valuation) If 
regular faculty ratings of students constitute a part of the 
first year pattern, record rating in Coluon 10. DESCRIBE RATING 
SCALE AND PROCEDURES OH THE COVER SHEW. "'^^^'^'^ »*ATIN6 



9-j 



,F«gt 3 

ROSTER tX)LUMN II> Regular dcpartwcntal exantnation< > (End of yor axntnation) 
% Record scort on tnd^'of^yeAr dvpartmontal oxMiQation in 

Coliaon II P«&t«l» fall • 0, or moro rofinod aeoro)'. 

DESCRIBE EXAHINATlMl, SCORING SYSTfM, ETC. ON 6mn SHEET* 



ROSTER COLUMNS (12) » (13)* and (U) ARE PROVIDED FOR AD HOC RATINGS FOR 
PURPOSES OF VALIDATION 'RESEARCH. tVo atandard rating aehadulaa for faculty 
ratinga of atudanta ara auggaattd for dofNirtMnta that do not asploy ragular 
rating procaduraa at tha end of tha firat yaar. If naithar of tho auggaatad 
aehadulaa la daaaad to ba appropriate, a dapertsant la aneouragad to daviaa. 
and apply a rating ptoeadura that It eonsldara to ba /ippropriata. THE* 
SUGGESTED SCHEDULES FOR FACULTY RATINGS ARE DESCRIBED OH A SEPARATE SHEET 
HHICK ALSO INCLUDES SUGGESTED PROCEDURES FOR OBTAINING RATINGS. 



ROSTER COLUMN 12. 



ROSTER COLUMN 13. 



ROSTER COLUMN 14. 



Standard Rating^ Schedule 1 (Rating relative to departaental 
atandarda and expectatlona) DESCRIBE PROCEDURES USED ON THE 
COVER SHEET. Only one rating to. ba reported for each atudant. 

Standard Ratln^t Schedule 2 (Rating in term of poCMtlal 
for advanced atudy In a field baaed on rater«*perceptlon of* 
general field denanda or requlrementa) DESCRIBE PR0CQ)URE8 
USED ON THE COVER SHEET. Only ona rating to be reported for 
each aeudant. 

Optional ad hoc faculty rating . DESCRIBE RATING SCHEDULE AND 
* PROCEDURES ON THE COVER SHEET. Only ona rating to be reported 
for each student. 



NOTE REaAR0JNC AD HOC RATINGS: Theae retrospective ratings should be baaed on obaervatioa 
of student performance In the departmnt from time of entry through time of the 
rating (ar time of student withdrawal from the department). Thus ratlnga for 
19 74 entranta typically will reflect observation over 2+ years of study while 
ratings for 197$ entranta will be baaed on obaervatlon over 1^ year% of 
graduate atudy. 



R05TER COLUMNS (15). (16), and (17) ARE AVAILABLE FOR RECORDING 
ADDITIONAL DATA ELEMENTS* 



ROSTER COLITIN 1$. Undergraduate Grade Point Average (DESCRIBE AND IDENIIFY SCALE) 
ROSTER COLVMN 16. OPTIOKAL DATA ELEMENT (DESCRIBE AND IDENTIFY CODING OR SCALE) 
ROSTER COLlTiN 17. OPTIONAL DATA ELEMENT (DESCRIBE AND IDENTIFY CODING OR SaLE) 
Ex.im;fles of adUltlonai data elements and suggested coding are as followat 

a) Degree objective rMaster's - !♦ Doctorate • 2) 

Use this code If department Includes both master*s and doctorate- 
seeking students and If first-year tasks and/or evaluation 
procedures are not comparable for the two groupa. 

b) Foreign student 

1 • Foreign student (English not native language). 

2 « Students for whom English Is native language. 
Use if foreign students are included on roster* 



e) TMr of bachelor's dogroo (rocord last two dltlts) 
d) YMr of blrtli\(lMt tvo digits of birth y%mr) 

0) UHdsrgrsdiMits Mjor ssm ss grsdiists f Isld 

f) tecolvsd uQdorgrsdttSto dsgrss fro« thls^instltiitlott 

nils institution ^ 1 ' * 

Othor institution • 0 

g) Quslity of undorgrsdusto institution ss Judgod by dopartasnt 
(dafins proeaduras for aaublishing ^'^ualioy'*! 

High 3 s 

Madiua « 2 

Lov or unknovtt • 1 

h) Hmhtd sutus 

Kajor f allovship or assistantshlp • 3 
Ofhar avard or typa of aid • 2 

Mo award or financial aid • 1 

1) IF ESTIMATBS OF CANDIDATES^ POTBMTIAL ARt MADS XOUTINELY 
AS FART OF THE ADMISSIONS PROCESS, THE RATimS, SCORES, OR 
CLASSIFICATIONS REFLECTING THOSE ESTIMATES COULD BE 
PROVIDED* THE TYPES OF INFORMATION USED IN ARRIVING AT THE 
ESTIMATES SHOULD BE DESCRIBED AS WBLL A$ THE PROCEDURES 
EMPLOYED* 



For aaeh dapartaaetal sMpla Involved, a. cover sheet should be coapleted* 
Space is provided for describing codef aqd identifying the data supplied* 
It is particularly Inportant that each datK^lenent provided be described (e*g*, 
nature of grading scale, rating prccpdures um, et^*)* 

If the supply of cover sheets and validity study roster foras is not sufficient 
please reproduce additional copies of the form* Additional copiea viU be for** 
warded upon request, however, if desired. 

^ All mterials when coopleted should be nailed as follows t 

Cooperative Validity Studies Project Call: 609-92l-90u0 for further 
c/o Kenneth M. Wilson, R 208 infornation or 

Educational Testing Service clarification of procedures 

Pr tnceton^ NJ 08540 Extension 2391 



In 
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CU Validity Study Project Uting rrocadWM 

SUCGCSTED lATtllC SCHEDUUS AMD FWCEDmiS 

A dtpartwitttAl faculty should ••lict tht rating •ehtdult that it to ho aoat 

.eoni^ottnt with Ito orlontatlon to tho aooootaont of atudont progrooo la tha 
dttartMnt. Only ona ratiag should bo roportod for oaeh otudaat. Katioff should 
bt basod on obsarvatlon of parfon»nco fro* satry to tlas of ratiag (or tUig of 
last offlcUl snrollssat, if sarlUr). 



Various procsduras^for ohtaialas ratiofs mmy ba eoasidorad. For axsaplot 

(a) Ad hoc dapsrtMatal coaiittoo to attlva at a f^eoasansus** 

rating for saeh studsnt listsd; consultation vith coUaaguas 
* rs casta not knoim to coanittaa Mobsrs or **dif f icult to 
assass** casss* 

(b> Solicit ratings of lista4 iodividugla frooi dspartntntal 
faculty ttsnbars. Bach faculty nssbar to rata aach studant 
vhosa raeotd is knoim* Ratings collactad and collatad for 
avsraging. A minimui of tvo ratings rsquirsd. 

RECAROLBSS OP THE PROCEDURES FOLLOWED, IVDEX HNALLT DEVELOPED FOR EACH 
SUDENT SHOULD BE ENTERED IN COLUWI^ 12, 13, or 14 of tha Validity Study Rostsr, 
depending upon usa of Schaduls I, Schadula 2, or a schaduls daviaad by ths 
participating department* 

PROCEDURES EXPLOYED IN DEVELOPING THE RATIHGS SHOULD BE DESCRIBED. 

Stan4a|rd Rating Schedules 
Scheviule I. Rating relative to departmental expectations and standarda 

Taking into account departmental expectations and standards, hov 

would you characterise this student*s record in the department? 
• 

4 • Distinguished ^ . 

3 • Cood to Strong 

2 • Adequate to Adequate plua 

1 «• Unsatisfactory to Marginally Adequate^ 

Schedule 2* Rating in terms of potential for advanced study in a fiel.' 

Based on your observation of this student^s performance how would 
vou characterize his or her potential for advanlied study In this field, ^ 
given vour perception of general field demands and requirements? 

4 » Outstanding performer; definitely qualified for 

doctoral study 

3 • Definitely master's caliber; probably capable of 

acceptable dc toral study 

2 • Adequate to adequate plus at the master's level; would 

not v*ncourage loctoral study 

I • Unacceptable or only marginally acceptable for graduate 
study at the master's level 

Revised 10/76 
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ETS USE ' 



CR£ nuttbtr 



OapattMntal eodt 



Imtltution 



VALXOin STUDY COVER SHEET 
DtpRrtMat 



Ooetorata 



• (City/St«t«) 

Nighttt dtgrM. off«r«4 within departntntt (1) , M >nfrU (2)^ 

ThU MApl* of flrot^tlM onrollad, full*tiM atudonto includoo (chock opplicablo otAtowmu)! 

(1) Only proopoctivo candidatoo for a Mator*a dogrt^ 

(2) Only proapoctivo candidatoa for a doct.oral dagroo 

Both proapoctiva maatar'a and proapactiva doctoral dagraa candidataa 

^3) foir whoa firat-^yaar program and avalqation procaduraa ara cottparahla< 

t4) for vhott firat--yaar ptograma and/or evaluation procaduraa ara not 
cosparabla. (If (4), coda dagraa ^Jactivt of aach atudant in 
Optional Data colum, aa indicated in inatruetiona* Slaborata 
on reverae eide of thia form*] 

PLEASE CHECK ( ) IN A BOX, BELOW, TO INDICATE THAT THE DATA EUMEMT DESICMATED IS 

PROVIDED FOR THIS SAMPU* 



ROSTER 
COa'MN 



Optional 
* coding 


Statuo 
code 


GRE scoi 
(scale< 


res 


First year 
sradea 


Regular 
department 
al evOT« 


Ad ho< 
departmi 

al rati 


z 

int- 
Ina 


Optional 
additional 
data . 


Sex' 
** 


Ethnic 
group. 


Statue 


V 
** 


Q 


Adv. 


GPA 
gen* 


Crit- 
ical 
GPA 


Rat- 
ing 


Exam 
acora 


Sch 
1 


Sch 
2 


Sch 


UGPA 


0th* 
ar 


0th- 
ar 


































U) 


(3) 


(4) 


(5) 


(6) 


(7) 


(8) 


(9) 


(10) 


(11) 


(12) 


(13) 


(14) 


(15) 


(16) 


(17) 



No explanation or further elaboration required if suggested validity study 
procedures have been followed* For noting exceptions to study procedures for these 
eleroencs and for describing other data provided* use the spaces balov* 
Please identify data elements described by column number * 



Rl)STF.R I 
COUMN 



! Uesoription of data provided* including description of procedures 
! scales of grade point averages, nature of regular rating 

procedures or departmental examinations, methods of obtaining 

ad hoc ratings, etc,) 



I 



flVFA 



ERIC 



In: 



Inttt 1 tut4un/iHf|>ar tnetit 



of 



J^gCS 



4,HK .VALU)ri*y . 
srt'DY HOSIKH 



optional 
Coding 



"IT 



tua 
Code- 



grades 



Regular 
de^attmsnt*** 
al aval 



5E 



Ad hoc 
depart«ent* 
aX rating 



Optional 
additional 
t dato 



Identification 



Sthnic 
group 



St^- 
tua 



CPA 



CPA 



Rat«* 



Bxaa 



Schtd 



SclMd 

3 



Sehcd 

3 ■ 



is: 



Op-, 



J2L 



Op. 



*. BRUEF DESCRIPTION OF SELECTED §TUDIES INVOLVING A »TWO-YEAR/ 
•SltlCLE-COHORT STUDY MOliEL ^ 

'* • - > 

*An «ffor« was mada during th« course' of the project tir^velop . 
a'SiuUl«>lnstitutlonal,i mult departmental study calling Jtot the 
concurrent participation of> the same set of fteoartments^ fro m several 
slallar insti'tations i? Several graduate schools with Established 
arrangements^forNharing data- on admlssions^^elated Questions 
expressed an -i^nterest Im participating in a study, based on a 
two-year stu'd^' aodel*' involving collecftion of data^on flrst-tJjne 
graduate students who entered Ph«D. oriented 'progriims in the 
Jepartments^of English,* Romance Languages* P||ilosophy, Economics * 
vieolJgy and Oeopnysical :>cienc^s, and Physics* respectively* in ,> 
fall lV74 ^ . 

€ ^ 

It was thought that^partidp/ition of a comnipn seC^of 
departments from several similar tftstftutions wcAild permit (a) 
the qotlectionof e9mbarable criterion data* beyortd grade 
averagis* subt^aa scores on comprehensive examinations and/or 
standard facuUT ratings find (b) analyses based on poole'd data - 
for the respective departmental samples. Information was provided 
oy several of the departments regarding their exaaiiiation practicej. 

There was- substantial variation ft the timing* scope* and 
coverage of tne examinations* a set b£ factors that t9iliated against . 
use of departmental examinations as a cpmmon criterion* Also* the 
goal of obtaining systematic facuicy ratings* according to a standard 
scnedule for students in the respective departments at ea^h interested 
scnool* was not realized* 

These conside tat ions, and the Inability of several of the 
•originally interested schools to provide data* effecUvely precluded 
development of^the study along the multi-institutional* oultl-depart- 
aental lines originally proposed. However, four institutions 
provided data for samples from five to eight departments. The study 
called for data for only one entering cohort. In consequftnce* 
sample size was unusually small*'" as indicated below: 

Institution A. N's ranged from six to 24 p^r department 

over six departments. 

Institution B. c<'s ranged from five to 19 in five departments. 

Institution C. A's ranged from four to 35 In eight depart- 
ments* with median N " 14. 

Institution D.' W's ranged from six to 66 ovcU six depart- 
ments, with median ■ 19. 

Ao 3ln»4.e criterion was common to all schrols and departments; 



<^«i oj; nor« prediccor ooservacions .were mtstttng tor one or Aord 
)Rics lo, nose: of tHe seaples., Indlyiduelized InecicucionAl 



4nd 
9 cud' 

reports were prepared for each of che four schools* tiowever» 
iacA of unlforalcy in- daca militated against the suwaarlsatlon of 
data across scnooJji* Results within the respective schools' were 
consistent wltn the geHeral propoisition that CR£ scotes should 
tend to be positively related 'to perftmmnce in graduate study* 

t'Unning and Implementing a study calling for, the Concurrent 
participation of ' a designated set of departments from each of ' 
several Institutions clearly posed considerably more complex 
problems than those Involved In planning and Implementing tne basic 
studies that called for institutions to submit, data for one or more 
departments selected on the basis of. local Interests ahd priorities* 



\ 
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^ F«f« 1 of 10 p«s«« 

CRE COOPERATIVE VAtXOm StaOtV 

XMCtuceloM for Cosplttlnf Validity Study SestMdt OaflnltloM 
Q md Frocadttr«s ^ 

X» D#£laltloM 

A. rift-tiaa EnrollVi Craduaf Studtmt : Hd graduate «ttidy prior to anrollmant 
In tha dapartaant. Sooa latltuda panlaalbla Cor tba iMlualon of lti41vlduala 
vlth llaltad pravloua traduata atudy. « 

B. Full^tima: StudanU daaa If labia aa ^full^tlM*^ atudaata aeoordlag to laatl* 
tutlooal^dapartaaotal erltarU at tlM of aatry. 

C* pa^raa'^saaklnat At tlM of adalaaloa, vaa eeoaldarad by tha dapartaant to ba 
a proapaetiva doctoral^dagraa caadidata. 

II. Procaduraa for complatlnt validity atudy roatara - A atudy roatar ahould ba eo»- 
plated for aach dapartMtit. laatructlona for coaplatlag tha atady roatar ara aa 
foUovat • 



BOSTS& COa^MH (1) 



Idantlflcatiott • Uat all fall 1974 antranta Mating tha daflnltlona 
outllnad abova* Hota that aaoa Idantlflcatlon la not required for 
purpoaea of the validity atudy and naaaa of atudenta aay be deleted 
froB any roatara prior to their tranaalttal to SIS. 

If Doncltlaena of the t7*S. vhoae Uck of flueaey In Engllah nay 
have eonatltuted a handicap In eoopletlng ORE requlresenta are 
Included they ahotild be Identified by special coding In Column 3 
(aee Inatructloa for that coIubbi» below)* 



ROSTER C0L13QI (2) Sey (option al coding) 



1 • Feoale 

2 • Hale 

ROSTER COLllOl (3) gthnie Croup Coda (optional^ but daalrable for validation research) 

U.S. atlzena 

I • Aaarlcan Indian. Eaklao, or Aleut 
a • Afro^Afflerlcan» Hegro* or Black 

3 «• Mexlcan-^Aaarlcan or Chicane 

4 • Oriental or Aalan^AMrlcan 

5 • Puerto Rlcan 

6 • Other Hispanic or Utln«*Aaerlcan 

7 • Caucaalan or White 

a • Other (disadvantaged) minority not classifiable above 
Hon*U*S* Citisens 

9 m *«Forelgn Student** (circle the code If English Is nacive 

language) 



*Two-Y«4r Model (Ph.O.-orient^d programs) 

MAy 1977 
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ROStER COLUtOi (4) 



For Mch acttdeoc li«t«<i. mxcod* Information ftardina p ra»r>«« 
in. th» d^^ontne m «pgtnt 1977. > ceordin« nlwfSlISm 

Cod* 

i • Contlnulnt th« d«p«rcaBat; en or «hMd of •ebcdulo la 
mtotlnt appXleablo dogroo roqulrttMnCs 

5 • Conelttulnt In th* dop«rta«nt; »Mwiwt b«hlad •ehodulo 
In aMClnt «ppllcmbl« dtgn* r«}ulrtMnct (no diseon- 
elwiltlos la •tc tn di n o no t eottatlns «tnMr ••••Ions) 



• ♦ - Continuing In tho dopartaont; •cm dl«eonelaultlM In 
•tCMdaae* aad dolaya la ■••elng «pplle«blo roqulrawnt* 

3 • Not conelnulng la tho dapareatnt; «uaalatlvft r«eerd at 
tla* of wlthdraval vm aadafaetoty 

2 - Kot continuing In tho dapartamt; euaulaclv*' record ae 
tlaa of wiAdnnral laeludad aoa* ladlcatlona of jub- 
^•eaada«d or aarglaal parfoiaaaea 

1 -.tot elaaalf labia abova (DMerlba en tba eovar abaae tba 
^ttaraa laeludad la chla eatagerx.) 

JSS '^'^ilirV^'^A.'SJrJ'lJiiiLr* "CORMHC CRB AmTDBE JM ADVANCED TEST SCALED 
rS; ^ i?^"' ^* STOOEHT. SECbSp THE SCALED SCORES OHLY (B.C.. 520 TSO 

SSeJ^nSS^ SSp^iivTS^lS" '!L°^ ^ recordxhg advanced testscaled 

(If S2?™^J^2?,213H MAJORITT pattern BT rjTIJIG M HELD ABBREVUTKM^ 
S™' STcSJuiSa,^* '*^ ^ ""'^ ^ FtltD) OVER THE 

V 

ROSTER COLUMN (5) CRE Varbal Sealad Seora 
ROSTER COLCMN (6) CRE Qtianeiuelva Sealad Sr»r« 

ROSTER COL'JMN (7) ,CRE A^vjiicad Taft - Raaaobar to write In field naaa abova sealed ' 
•core eatrr for aU axeeptloaa co the aajorltr Advanced Taat flald. 



ROSTER COLUMN (8) 



ROSTER COLUMN (9) 



PnderKraduaca Grade Point Avtraaa (VGPA ) (optional, but daalrabla 
If #vallable2. - Eater the UC?A In Coluan (8) aa nornally eoaputaid 
aad ^ed In tha adadaalena proeaM. Tha aeala aaployed aheuld be 
daacrlbad oa tha cavar ahaat. ^ , m 

Adaiaalona Radnt or Ranklnt (opt ional, but desirable if ayallabla 
for all or noat atudenta) - Enear »«v rating 
reflactiag aa adaiaalona-related aaseaaaant of relative potential 
or pro^ae. If for exaaple admitted appllcanta were claaalfled on 
tha basis of their adalsslons credentials, tha "ranking" Involved 
should be entered for analysis In relation to the criterion varlablea 
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r«pert«d. FROCEDURES USED IM'AWaVIHC At 181 ADKMSIOW UTHK 
SttOOLO M OESCKZIED ON tHS COVOl SHECX. 

tOSTEll COUnOO (10) THIOOCa (U) AlUt to it USED TOR ^™ '2!*.,S^SIJSI A 

•^SeS" FOR EACH StOTEW. EAd FAHtXClFAtlMC OEFARtMEHt 18 EHCOOTACED »WWXDE A 
cSSSiStVE CRAdStE «Se MWt ATERACB AMD AM AVERAGE "COMSEMSUS^'RAtW; » ^WDW* 
SiSoSoSl^RDXMO to A SCMEDOU OOttlMEO " «^J^<2?S5^ 
MEASUU tS MESDED tO COMDUC* A StOOt. EACH W;»S^^S«?S52«^!2lS^ 

8C8E0UIE0 FOR tHE FtRSt AMD/OR 8EC0MD TEA* "S^-JS/SSL^KI 
STOTDItS All lEGOIARLt WAPt AFtEl CMS OR tWO YEARS OF StBOt. tHOSB RAtXMOS Wt >B 

RXFORtEO. 



}U)StU COLtMM (10) 



ROSTER COLUMN (U) 



ROSTER COLUMM (12) 



ROSTER COtaW (13) 



eu»ul«tlir« Cr«d. .>*. ermd« Point Ai>>r«rt fCPACUM) « "^"{.S^^ 
pelat avMai* b«««d on cours. work co^i«tto aurlof th« flrtt wo 

of triduat. study, or «U wck coivl^ttd P'i*!* 
SSdr««.l.fro. tbo d«p«rtii«it. l£ .wllcaklt. OBSCRZM GRADE 8CAW 
AMD AVIRA6ZM6 RULES OK tBB COVEH SHESX. 

Ad tee RAtlot of Stttdon t P^rfarMneo Rolatm to DfyrtamtAl 

MTratla^ ior irarpoi««lf vmlldlty ttudy, th. r.tliit» /hould b. 
bM«d on obAorvatlowi of porfotMnco lit. tho dopartMnt fro« toao 
of otttry to ela* of rAtlng for cnrrontly wirollod itudAats, •»« . 
froa tim of ontry to timm of iiltlidr«w«l for othom. 

A ttwdard rating ichodulo 1« attacfhod. V of tho ichodulo out- 
Untd In th« AttAcluwnt 1« oncouTAfod. Hoi»«»«r, If , 
•d hoc proctdttTO It doo«d aoro approprlAto, a d«p«rta«t should 
l««l fro* to thAt pXOCAdUZA. 

"Seora" on Critteal r^i« mttfm (oMMlit^tnm. MOortK PtoflelAOcy). 
^tor horo A lunoAty Wa rof iocting a itSdont'i p«tor»Aneo on 
tho first critical «Ai«lnAtlon thAt AwbAM of An 

«y b. CpACfd to h«VA .tt«ipt«l •l-^-S ^^J^y^irJ'f 

T«Ar of tTAduAto Study* to bo consldorod "crltlcAl" tho oss^ua- 
S:i(:5 uZT^i J7b. «t by All .splr«i« 2 •JSSJJ^oJ'S!* 
At leMt P.../FA11 Aiid pr.fwr.bly . mot* "«i?^.S2SS?Sncf) 
fonunc. .hould b. r.port.d* Th. nAture of th. .««tnAtlon(s) 
lnvolv.d should b. doscrlbwl on th. cov.r sh..t Along with A 
dMcrlptloa of th. scoring And th. scorns roportod. 

Ra «ul«r FacuI -T stt.d.mr P.tfonnAnc. - Rword in thU 

coSmn tAtings of stud«>t p.rfo ««nc. thAt «Ay hAv. b..n "J* 
llVZl otZ first or s.cond y«r of .tudy. a. p.rt of th. nemAl 
or rliuUr Stt.m of d.pAra-nt.1 procdur... Th. tladng of th. 
StISrtl.g : IS of fl?.t y.Ar) And th. procdur.. •«Ploy«J •» 
^11 a. th. pUc. of th. rstiag. in th. totAl pAttom of dopArt- 
oMitAl r.qulr«.nM should bA doscrlbad. 
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m USxSlSS* 0' SUCH MtA, AMD SUGGESTED COoSc piSSS 

•) Ytar of baehtlor^t d#tr«# (Imc two difita) 

b) Ttar of birth (laat two digits) 

c) Undorsradwto Mjor 9Mm m graduato fiald • 1, othar • 0 
4} Kaeaivad uadargraduata dagraa twm thia laatitutioo • 1. 

othar • 0 • • 

Kaeaivad uadargraduata dagrjia froa highly aalaetiva 
inatitutiooa (a*g*, DUAKTS • 1, othar • 0) > 
f) Avard autua ^ 

4 - Rolda or haa hald Mjor raaMreh faUoimhip or 

aaalauotahip 
3 m Holda or haa hald aajor ooaraaaarah ftllonahip 

or Maiataauhip 
2 - Holda or haa hald aaaaiogful but not major . 

fallovahip or aaaiataotahip 
1 - Holda or haa hald ao typa of faUonahlp or 

asaiataatahlp 

II IS OffORIANT TO DESCRIBE lACK OWIOKAL MXA ELEMEHT IWLl. 

taporunt that Mch d«c« •Immoc provided b« dMcrlbtd ». .eill. cp! ST^Ikw! 

AU aattrlala whaa eosiplatad should ba aailad aa foUoira: 

Cooparativa Validity Studias Projaot 
c/o Katmath M* Vllaoa* R«»208 
Edueatloaal Taatiag Sarvlea 
Friaeatoa, Nair Jaraay 08540 

CaU: 609-921-9000 for furthar inforaation 
or clarlficatloa of procoduraa 
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Amchatttt 



CU COOFEIUIXVS VALXDITt STUDY 

tattruetiuot for Co«pl«cinf RAtiags: lUtiat Ptoe«dttt«s 

A 

^ ttcowptndtd Standard Rattni Schtdult 

It « d»?4rt9ttnt ftXtcts to dtvolop ad hoc ratings for purposts of GR£ vmlidaeion rtSMreh, 
It is rteocmndad zhAt cht ratlagt be d«v«lop«d ibeordlag to tb« folloviag tehtdult: 

% 

Taking into account dapartmantal ^actationa and 
atandarda* hov vould you eharactariaa tha atudaat'a 
racTord in tha dapartaanc? 

4 • OiatlAguiahad 

3 • Good to Strong 

I • Adaquata to Adaquaca Plua 

1 • Onaaelafactory to Marginally Adaquata 

JUclngs aay ba obeainad by varloua aaana. For axaaplat 

a) By achieving at laaat ttfo indapandant ratings 

for aaeh atudant which can than ba avaragad~a.g., 
hava all faculty naobara rata aach ttudant known 
to than; obtain rating fros a atudant* a advisor 
plus ona additional faculty aanbar, ate. 

b) By having an ad hoe dapartaantal coaaittaa davalop 
a **conaanaua^ rating for aaeh studant. 

Ragardlass of tha procaduras followad, tha avaraga rating or tha conaansua rating daval«» 
jpad iot aach studant should ba antarad in Colxan 11 of tha validity study roatar. 

Procaduras asployad In davaloping tha ratings raportad should ba daacribad on tha covar 
saaat* 

NCTS: **ai8bars of tha tntaring cohort %iho ara no longar anroUad in tha dapartaant 
should ba ratad on tha basis of parforaanca during their period of anrolloant 
If « In the judgnant of faculty lattabars involved, a ratable pattern of attainaeant 
vas astabllshed during that period of anrollmnt. 



3' . 
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^ for ETS V Oni^ 
CW {tumbtr 



VALZOm STWY COVU SUCET 



Iiuitltutlon ^ 0«p«rtMtit 

USS) <aty/Se4tt) 

HtghMt dtt'M off«r«4 viehU 4«p«ttMiit: (1) M — f (a) ' D ^ccarmf 

ThU tMpXe of <ir«e«elM Mroll«4» fttll«>tiM studtfttt iMlu4#i^ (elMefc «ppXi«abU •UMmau): 

C2) Onlf prospMtlw emdidACMi for « 4oc(or«l doftM 

Both prospaceivt sMttr*« and proap«etlv« doctoral datroo eandldaeaat 

^or vhos firae-yaar pvograM and avaluatlOQ procaduraa «ra eraparabla* or 

^or wHott flr«t«yaar profraaa and/or avaluatlon procaduraa ara not coapmrabla* 
(If (4), coda dagraa objactlva of aaeli atudaat la Opeional pau colum, I • 
HMear*a» 2 • Ooceoraea*) . 



?llASt CRSOC ( ^) IH A BOX ItLOW TO IMDICATX TBkt THE Wtk SUME» DCSICSmD IS flOnOSO 
FOR THIS SAMPU. 



Sax 


Coda 


rt$$ 


""31— 
VEAa 




AOV 


OCFA 


■ 


i»r" 


fill 1 IW 

^ ho< Crle 
SacTl Exam 


Rata 


3ptlor 


- jA . 

Optloit 


3ptioii 


)pelo{^ 






























t 



}fOXtz In :ha ipacaa baIoir» provlda Itifbrnaeion rtquirad eo Intarprat aach of eha dau 
tiamanea chackad abova* Whara ataadard codiag la provldad, oalf axcauioaa to 
chat coding aaad ba daaetibad* 



ROSTZR 

(2) Sax (FaaaXa • l» Hala • 2) 

C3} Ethnic Croup Coda (par Inatructlons* standard) 
(^) Prciraaa Coda (par Inatraetlona* standard) 
(5) ORE Vtrbal Scalad Scora (standard) 
vo^ ORE Quantltaclva Scalad Scora (standard) 
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cotvm 

(7) CRC Advanctd T«»t $cor« - FIms« ll»t th« tMt tltlw (fltldt) for vhlch Ust fior** 
«r« «valI«bU* Also show th« co4« or abbrovUtlon vlth vhleh tho to«( Is Idontlfiod 
on thm validity study ro«tor, £SS;s«* Soo instructions for CoImm (T). 

Msjoritr (or only) fioW ' 

Second flsld . 

third fisld . • <€) 

Ochor ! W> 



(S) Indoftrsduato Crsdo Point Avtrsfs ^ Otscrib* ovnraU cu^Ustivo, uppor division 

cumXstivs, Mjor fiold only, senio Mplctysd, otej. 



(9) Admissions ting or Unking - Ooscribo horo th* rank, rating, or eospositn seoro 
^ (s^g** 2 > 25 UGFA) v%j^tfd. 



(10) Cuaulstivn Grsduats Crnds Foint Avnrags - Doseribs seals* asthod of eoapucntion (••g** 
hoM bours in dnnoainstor art aeeumlatad) • 



ai) A4 Ifec Oapartwintal Rating (standard) - Obscriba how rating was datarvintd (a.g*. 

bv avaraaina two or nora indapandant ratings, by consansus procaduras, typical nuabar 
ot ratings involvad in avaraga, ate J USS SE?ARAt£ SK£CT IF STACt HERfi IS HOT 



N. 



ftOSTER 9 F«S« 3 of 4 

COLUHN * . 

Crlclcal £x4«iMeloQ **Scort" • la thts« «p«c«s prcfJ^« a brl«f d«terlpclon of Ui« 
•)U«ln«tlon(s) Involved* and a translation of acotaa raporcad, if nuaa^ln (a*f*, 
Hxgh Paaa • 3» Pais • 2, Quallflad Pass • 1» Fall • 0)* Uaa a aaparaca ahaac to 
daacrlba tha axamlnatlon proeaaa wra fully. tiOTCt IF A OCPAJIIHCKT HAS PKCVIOUSU' 
PROVtOCO A OCTAILCO OCSCItXPTIOH OF TBC UAMIHAXXOM PlOCSSSt TXU XTIM KCCO Kot IC 
COMPLETSO. 



(13) RatuTar Faculty Rating of Studanta * Oaacriba hara tialng and purpoaa of ratings, pro« 
caduraa uaad» ate* Ptovlda lafocaatioft raquitad to ttanslata nuMtic codlni** 



(14) Optional Oata UaMot (dascriba fuUy) 



(15) Optional Oata CXattant (daacrlba fully) 



(16) Optional Data Elaaant (daacrlba fully). 



vl^^ Opti^onal Data Elaaant (daacrlba fully }^ 



Inforracion About cha Adalsslons Procatt trtd Scudane Prcarftss 

(optional, but highly daairabla if availabla) 

• *, 

antwar cua followlnti quaaeions baaad on studiaa that may hava baan dona in tha dapart- 
aane or '*batt ateifflataa" If no studiaa hava baan dona or if statistics ralatad to tha quaa- 
tlon hava not baan ragularly aaintainad. Tha quastions partain to individuals lika thosa 
Includad in tha praaant saspla (a«g«, not **trans£ar** graduata studanta)* 



I. What pareantaga of applxcanta typically la adaxttad? 
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2. What p«rc«ntagt of adalsslons offtts typically is accopttdt 

U. Civt your btse tseinieo of th« ptretnust of firse*eitM ttaduaco tcudones in • 
. givta cohorc likoly eo ^onpUeo Ph.O* rtquirouiMU la tho dopurtnooe* 

ib. What p.trctnta«t of an tnttclng cohort it likely to fail to nuljfy for Ph.O. 
candl<Ucy witHdraw with unaatltfactory racor4 in cotirtc wonc, failura 

on qualifying axaainatiooa, failura eo attaspt apadflc raquiraaants on 
achadttla)? — — 

3b« What porcantaga of an antarlng cohort ia likaly to laava tha iapartnant without 
a dagraa, but with a baaically satisfactory racord of parforssnea (a*g*» no 
••failura** to naat axpliclt ra^uiroMnt)! • . 

4. What ia tha bast aseiaata of madian yaara to tha Ph.D. , »»triculation to dagraa 
cottfarraU for firsc^tiaa antrants Uka thoaa includa4 in c^a saspia, who €o«- 
Plata 4agraa raquiramanta in tha dai^artaant? 
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Appendix C 



C-1. Illustrative study report 



^1 



C~2. Tabular summary of selected study findings 
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Appendix C-1 

Ul\i8trative Study Report 
GRE COOPERATIVE VALIDITY STUDIES REPORT 



Educational Testing Service 
Princeton, NJ 085A1 



Validity Study Report 



for the Departments of 



BOTANY /MICROS lOLOGY 
GEOLOGY/GEOPHYSICS 



in 

Graduate School A 



May 1979 



Sponsored by the Graduate Record Examinations Board 



AppendlJt C-1 ^ * : • 

^ XNIIODUCTIPH 

This report if^s b««n prsparid «• part of th« Coopttr«tlv« Validity StudiM 
?roJ«et baing conduetad by EducatioaaO. Tatting' Sarvica for tha Graduata Raeord 
Exaainationa Board (G&E). Tha projact ia daaignad to balp graduata achoola and 
dapartaanta ganarata up-to<-data avidanca ragairding tha Xavala and pattafaa of 
ralationahipa batwaan GRE aeoraa (and othar data uaad in adaiaaiona, ^ueh aa ^ 
tha Ondargraduata Grada Point Avaraga or UGPA) and ona or aora liaitad but 
ralavant aaaa^aa ^f parfor^ca in graduata aehool during tha f irat yaar of 
atudy (a.g., critarion aaaauraa auch aa a*GraduAta GPA, faculty ratings, or 
departaantal exaainationa) , in racaptl/ .anroUad cohorts (1974 and 1975 
antranta) of firat-tlaa anrollad, full-tiaa, dagr^ia-aaaking students. 

G&E validity study findings for 'tha dapartaantal saapla(a,) for which your 
institution provided data are prefe^nted herein. ' Inforaation ia provided 
regarding the level, patterning, and 4istribu'tion of acorea for the aaapla(s) 
on each of the variablea for which data were subaitted. More specifically, 
aeaaures of central tendency (the arithaetio average or aean) and. variability 
of acorea (the standard deviation) |re repprted. ' 

Evidence regarding ^e relationship be twain each ajdaiasions variable, or 
predictor (such as the GRE-Verbal, GRE-<)uantitative, or UGPA), and each per- 
f oraance or criterion variable (such as' a Graduate GPA) ^a presented in tenia 
of the coefficient of correiation, a generally faailiar index of association or 
covariation between variablea. The aizef of a coefficient indicatea the degree 
or closeness of association between two variables on a scale ranging 'froa .00 
(indicating no relationahlp at all) through ^ 1.00 (indicating either a perfect 
positive or a perfect negative relationahlp). A positive sign indicates that 
higher standing on a predictor tends to be' associated with higher standlhg on a 
criterion variable, whereas a negative sign indicates that higher atandlng on a 
predictor tenets to be aasociated with lower atanding on a criterion. When 
used to express the relationship between predictor and criterion oeasures in 
adolsslons settings, observed correlation coefficients (known as validity 
coefficients) are almost always positive. In GRE^validlty studies, for exaople, 
coefficients between .20 and .30 are typical, for individual predictors (c^. 
•table 1, p. 5). 
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SoM Liaitlng ConiilderAtiOM 

At yen wminm this report, it la quit* loportant to 1ui«p In mind the 
liait«d scop« o£ th« validity assassMot that ia baing undartakan in thaaa 
Cooparativa Studiaa. thay focua primarily on firat-yaar parformanca criteria 
and thay kra baaad on qulta anall aanplaa in moat inatancaa* Sttall*aaapla 
aaalyaia in any contaxt calla for-fara in tha intarpratation of raaulta, of 
couraa. Addad intarpratational conplicationa ariaa in aaall aanpla GRE validity 
atudiaa- (a) ainca tha diatributiona of potential pradictara (e.g., GEE Aptitude 
Teat acorea and UGPA) tend to be raatricted in tange becauae they were "uaed" 
in the reoruitaent/aelection pro^eaa and (b) ainca local "yalidity atudy ^ 
noroa,'* derivable only through periodic replication of atudiea, nay be either 
lacking or inadequately develope<l in many departmental aattlnga* 

In evaluating validity atudy findinga generally it ahould be recognized 
that **predictive validity** 'la alwaya relative— validity ia not an abaolute 
quality of a GRE teat or any other admiaaiona variable. A validity coefficient ' 
ia aimply an index of the relationahip betveen a predictor and a given criterion, 
both of which are leaa than coo^letely reliable meaauraa, in a given prediction 
context and aampla. The validity coefficienta reported herein, therefore, apply 
only to the apecific criterion neaaure(a) employed and the 8a8q>le(8) or group(a) 
In which they were determined. Reaulta ahould not be generalized to other 
criteria or groupa. 

Queationa regarding the predictive validity of GRE acores for partictilar 
subgroupa muat be addreaaed in appropriate aamplea of the varioua aubgroupa of 
Intereat— e.g. , ''older'* atudenta, '*tranafet** atudenta, or ''minority'* atxidents. 
In the present series of studies t the focus Is on ^f lrst«<»ti]ae enroll^t full-* 
tlaet degrees-seeking students.^ Determination of validity coefficients In more 
refined subgroups Is not feasible due to the small size of the sample(s)a 

It is Important to recognise In passing that the flrst-^year criteria under 
consideration in this study (such as Graduate CPA or end-of«»year ratings) are 
themselves subject to empirical evaluation as potential predictors of longer-^term 
crlVerla of ^^success** In graduate programs (Saga^ what Is the observed validity 
of flrst-^year Graduate CPA for predicting graduation versus nongraduatlout or 
performance on PhaO* qiial^fylng examinations?)* 
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Moroatlvs f«rtp«ctlv« and Iat«rpr«t«tloiMl 

Evld«ac« £ro«i^v«lldity scudlM th«t havs b««ii conducted la « v*tl«ty 
of •ducatiooal •attloga, axtanalvaly la uadargraduata and Uw schools and Isss 
axtanalvaly at thm graduata laval, as wall as avldanca o£ tha posltlvaly 
IntarraUtad organisation o£ huaan abilitias, suggasts that validity coaffi- 
Giants for acadanic pradictors such as tha ORE and HCPA and acada«ic critaria 
such as Graduata CPA oi faculty ratings should tand to ba positiva* In assanca, 
it is raasonabla to assuaa that individuals with "bat tar qualifications** (as 
raf lactad In thair past acadamic racord and thair scoras on varbal and quantl- 
tativa ability maasuras* for axanpla) should tand to ba sonawhat *'battar 
studants" (as rtf lactad in faculty assassmants of thair work in graduata 
coursas, saminars, and tha lika, for axanpla). 

If nagativa coafficants ara obsarvad in validity studias, thay usually ara 
-mil and nay ba axplainad as falling within tha nornal ranga of axpactad variation 



dua to sanpllng arror ar<iind a "charact aria tic" or •population" valua that is 
low but probably positiva. Nagativa coaff iciants for acadamic pradictors and 
acadaaic critaria ara, tharafora, proparly parcaivad as thaoratically anomalous. 
Whan obsarvad, thay indicata tha naad for furthar axploration and analysis 
dasignad.to illualnata tha particular circuastancas involvad. 

It has baan astablishad that tha sita of validity coafficiants tands to 
vary Invarsaly with tha dagrac of rastriction of ranga o^ talant in sanplas 
balog studiad. Tha intarpratation of obsarvad validity coafficiants for ORE 
scoras or UCPA is aspacia.lly conplicatad by tha fact that graduata atudants, 
ganaraiiy, raprasant a highly salact group with raspact to acadanic ability and 
past parforaanca. In dapartmantal sanplas such as thosa Involvad in validity 
studias, futthar rastriction of ranga on thasa variables is Intrbducad aithar 
directly (whan GRE and UGPA have baan used in selection) or indirectly (when 
other related variables nay have been* used). Rastriction in range on one or 
more of the predictor variables under consideration nakes it difficult to 
obtain a clear assessment of the actual "value" of the predictors involved 
since observed validity coefficients tend to be lower than would be the case if 
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« full r«ns« of c«l«nc** (•.!•, a group roprtsontaelve o£ all eollego taniort) 
could ba ineludad in dapareaaatal airaplaa.* 

RaatrictloQ ia eha ranga of eritarlon valuaa alao eonplicataa eha Intar- 
prataclooal outlook, ff crlcarion valuaa, auch aa Graduaca CPA, vary only ovar 
a vary lioicad ranga (a.g., A or B) dlffarancaa in acivlaac parformmca aay uoc 
ba aaaaurad rallably aad thia alao tanda to load to uadaraatiaation of tha 
ovarall utility of a pra^ictor. 

« 

SoiM avldaaca ragardlng tha charactariatlc lavala of validity coafflclaata 
that hava baan obtainad in rapraaantativa atudiaa involving G&E and UGPA and a 
Graduata GPA critarion la providad in labia 1 which ahowa aadian coaff idanta 
£ro« a nuabar of atudiaa, aa racantly auaaariaad by WilUnghaa (1974).** Alao 
ahown ara data fro« undargraduata validity atudiaa involving a GRE-cooparabla 
aaaaura, naialy» tha GEEB Scholaatic Aatitu^a Tm^ (Varbal or SAI-V and Hatha- . 
aatical or SAI-M) and collaga fraahaan-yaar GPA (Schradar, 1971). Thaaa lattar 
data indicata hov validity coaff icianta tand to ba lonar in aaaplaa that ara 
highly aalactad on varbal ability than in aaaplaa that ara mora rapraaantativa 
with raapact to varbal ability. ( 

In tha graduata school atudiaa, aadian vattSitiaa for cha GRE Aptitti^a 
Taat conponanta baaad on atudiaa involving GraduaO^PA critaria in aatolea 
froa a variatv of diadnl^if wara alightly highar than .20, ehoaa for tha GRE 
Advancad Taat or tha UGPA, alona, wara about .30, whila tha baat-waightad 
(i.a. , Bultipla ragraaaion baaad) coabinationa of GRE Aptituda acoras and UGPA 
yialdad validity (aultlpla corralation) coaff icianta avaraging around .45. 
Nota that chasa coaff icianta ara aiallar In pat tarn and laval to thosa obaarvad 
In undargraduata sattinga in which aaaplaa wara highly rastrlcted with respect 
to SAT Varbal scores (i.e., high aaan «nd aaall atandard deviation). 



*In recent year a, GRE-Varbal scoraa for candidatea nationally have had standard 
devlationa of approxinataly 125, and tha standard deviations of GRE-Quantltative 
scores hava been approxiaataly 135. In departaental aaaples such those 
Involved in the preaant atudiaa, standard deviations of 75 to 90 on one or 
both these variablea ara not uncoaaon, indicating that the range of ability 
available for study is considerably less than that in the total group of 
Individuala taking the GRE Testa nationally. 

**Warren W. Willinghaa, Predicting auccess in graduate education. Science . 183 . 
pp. 273-278. This is a brief but comprehensive overview not only of represen- 
tative GRE validity study findings during 1952-72 but also of basic validity-study 
concepta, probleaa, and issues. * ^ 
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ItedUa Validity Co«££lcl«att and lUng% ot CotfflclMts Itt StudUs 
o< Coiv«rabU Ft«dletor« aad Cradt 4vtr«ta Crltarla la CrtdMata 
Md Uadargraduata Sattltttt 



fradletora m 
aporoprlAtt 
to Itvtl of 
s^udy* 



ate studios 
U trsdusto 
school 
sottlsgs 

Mai 



CUB UT-vslldlty studios in 
uadorgrsduato sosj^los vhleh voro 



Hlftb 4 howtMMus Homogsaous Uprosoatstlvo 
oa SAX*?** «i SAt^V*** 8AT-V seoros* 



Mda. (rsaio)f Mda* (fsngo) Mda. (rsajo) 



CKE-V«rb«l (Cr«d.> 
SAI>Vttrb«l (U.C.) 


.24 


(46) W 


.22 


(.11 to .44) 


.31 


(.13 


to 


.46) 


.39 


(.26 


to 


.54) 


SAT-Hatb (U.C.) 


.23 


(43) 


.24 


(-.01 to .46) 


.27 


(.11 

• 


to 


.40) 


.33 


(.20 


to 


.48) 


C&£ Advanetd (Cr«4.) 


.30 


(2$) 






















Oadtrgrad* CPA 
aigh School K«eord 


.31 


(26) 


.40 


(.32 to .37) 


.44 


(.26 


to 


.59) 


.55 


(.33 


to 


.67) 


caC ■«> UCPA (Crad.) ' 
SAT ■»> HSX (U.C.) 


.45 


(24) 


.46 


(.33 to .61) 


.32 


(.34 


to 


.66) 


.62 


(.46 




.73) 



Sotot 



Cr«du«t« school dAt« «ro froa Wlllinthaa, W.W., Pr«d4ctiat i» , 

,r.d«*t. .ductioa. jlslissi. iSi. "3-278. T.bl. 1. J«d.r8r«iu.t. 

validity d«t« art from Schrador, W. »., Tho prtdictivo validity o« 
CoUaga -Board adalaalona taata, la Angoff , W. B. (Ed.), Tftf V9UWf 
i>n^rd\d«ia.lo. .. taati«« oroaraau (Prlncaton, HJ» CoUaga Eatraaea 
Esaalaatloa Board, 1971), pp. 117-146. 

nha coaffldaata In colum 1 o£ tha tabla raflact v*!"^'^** 
CBE-Ouantltatlva, C«E Advaacad, Uadargraduata CPA, and CBE-CCPA eoapoaltaa, 
teapacttvaly, la graduata achool aaaplaa. All tha ramalalng coafflclaata 
raflact valldltlaa for SAT-V, SAt-M, tha high achool racord, and SAt-lSa 
eoapoaltaa, raapactlvaly, la aaoplaa o£ eoUaga fraahwa. 
••Studlaa la 18 aamplaa o£ havlag aa SAt-Varbal aaan abova 600 and atandard 
davUtlon o£ 65 or laaa (uadargraduata fraahaaa). 

•••Studlaa la 95 aaaplaa o£ uadargraduata aaa and iMMn fraahaan havlag SAt-Vatbal 
standard davlatlon o£ laaa than 75. Madlan valuaa raportad MPf^wly »•» 
and womin by Schtadar hava baan avaragad for prasantatlon In this tabla. 
#Approxlaataly 80 patcant o£ tha obtalnad coafflclanta wara within tha ranga 
apaclf lad. 

#jNuab«r o£ coaffldaots upon «hlch aaeh nadlan la baaad. Studlaa ara aum- 
aartzad without ragard to tha flald o£ atudy Involvad. 
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TAbU 2 

V*rl*tlon la Validity Co*£fici*ati for $«l«cttd Fredletors io Studlas 
teplicattd AeroM «nd Within Un4«rgr«du«ttt ColltgM 



C0llttg«/ClMS 



Collttgtt A» 1966 
1969 
1970 
1972 

College B, 1968 
1969 
1970 
1972 

College F, 1968 
1970 
1972 

College J, 1968 
1969 
1970 



typical 
Maple else 



(332) 



SAT-V 8AT-M leak Achievement 

test ever* 



(266) 



(33) 



(62) 



.16 


• 13 


*.27 


.32 


.3Z 


.23 


.37 


• 37 


.31 


.28 


.32 


.52 


•30 


.20 


•28 


.35 


.20 


.21 


.43 


.42 


.29 


.11 


.39 


.37 


.27 


.05 


.31 


.22 


.21 


•.06 


.20 


.24 


.23 


.13 


.42 


.31 


.25 


-.05 


.21 


.43 


.28 


.13 


.39 


.24 


• 27 


.49 


.61 


.43 


.20 


.33 


.47 


.46 


.48 


.54 


.54 


.56 



Kenneth M. Wilson, The contribution of neestires of aptitude 
(SAT) and achievement (CEEB Achievement Average), respectively, 
in forecasting college grades in several liberal arts colleges. 
Research Bulletin 74-36. Educational Testing Service. 1974. 
Table 4. . 

These findings suggest that prediction of Graduate CPA from GRE scores or 
Undergraduate CPA might be accomplished at about the same level of "accuracy" 
as that involved in predicting college freshnan-year CPA using comparable 
predictors in "high ability-low variablity" undergraduate settings but at a 
lower level of accuracy than that found in more representative undergraduate 
settings and samples. 

Some indication of the range of coefficients obtained in replications of 
validity studies across colleges is provided in the "range" data shown in Table 
1. Table 2 provides illustrative data showing how coefficients may vary under 
conditions of replication within a college as well as across colleges. The 
coefficients shown are for samples of undergraduate women in successive classes 
in several selective colleges (Wilson, 1974). Among other things, these data 
suggest (a) that "small sample" studies do not necessarily yield less stable or 
Interpretable results even though the potential for marked variability due 
solely to sampling error is greater in such samples, (b) that regardless of 



o 
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•«spX« lnf«rac«s regarding **tbm validity" o£ « glvaa predictor or tho 

ralatlva validity of aavaral pradletort «»lelil& a givan pradlctlon cootaxt 
ahottid oot ba draim on tha baaia of ona validity atudy, and (c) that tha lavals 
and pattana of validity coafficianta that obtain for ona aducatlonal aatting 
do not nacaaaarlly hold for anothar* 

Quastiona of *Valghting** 

In validation raaaarch ganarally, it haa baan found that an approprUtaly 
vaightad coabination of two or aora avallabla pradictora fraquantly yialda 
highar validity coafficianta than any of tha pradictora conaldarad aaparataly. 
Thia pattern la auggaatad, for axanpla, by tha GRE validity atudy findinga 
suaoariaad in Tabla 1. Conbining GRE Aptitude acorea and the UGPA tanda to 
reault in ioproved validity. ^ 

In large-aaaple validity atudiaa, queati^a regarding the noat effective 
weighting of two or aore predictora with reapect to a given criterion tradition- 
ally have been addreaaed by application of appropriate attltivariate atatiatical 
aethoda, principally aultipla correlation and regreaaion analyaia. One typical 
outcoae of theae atudiaa haa been the developaent of "equationa" for obtaining 
a ••predicted criterion acore^^ for each individual. Weighta indicating the 
relative contribution of two or aore predictora in auch equationa are apedfied 
In the analyaia. In aaall aaaplea, however, multivariate procedurea have aore 
limited operational utility dwa to the fact that the reaulta obtained (the 
weighta derived) may reflect too doaely the poaaibly idioayncratic pattema in 
a email aample ( ••overf itting the data^^) and thua vary markedly in aubaequent 
small samples. 

Given the typically amall aiae of the aampleCa) under consideration in 
these GRE Cooperative Studiea, and the additional limitationa impoaed by the 
fact that particular observations may be lacking In some instances (e.g., some 
Individuals In the sample may not have GRE acorea, or a criterion oeaaure) , any 
appilcation of multivariate analysis that may be reported herein is intended 
only to facilitate limited conaideration of certain of the principles, advan- 
tages, and persistent problems involved in developing reliable information 
about the relative contribution of predictora and in combining two or more 
predictora in admiaaiona aettinga. 

All Compoaltea of GRE Scorea Require Empirical Validation 

It Is particularly important to call attention to the potential hazards 
involved In uaing any ••intuitively appealing^^ procedure for combining GRE 

1 "^r 



•cortt that hat not ittn cArtfully tvaitMetd tnpirically* Ott of « GK£*Tot«l 
(«. slopl* addltivtt cottbinaeloa of cht GRC-Vttrb«l «ad CRC-Qu«tttit«tivtt scorM) 
coMtitutM a higbXy rtlavut Ult** AxaBpltt of « proeoduro th«t thotad b« 

«void«d In tho absoaeo of •apirlc*! ovldonco that GRE-TotAl is noro v«Xld than 
•ithor of its coapottftntt* la cortain drcitmaeancM not infroquoatly oaeouaUrod 
la practico C>E«»Total actually ±m !«■■ valid thaa GRE-Varbal or GRE-Q(«aatitativa 
oaly, whichavar haa tha highar validity. 

To Uluatrata earuia approachaa to waightiag, aa wall aa abaa of tha 
problaaa iavolvad, cootidaration ia glvan to tha validity of varioua coapoaitaa 
of availabla pradictora. Aaaia it ia important t a atraaa tha fact that tha 
pumoaa of doina ao la yi^^p^i^Yf iv didactic . 

A Briaf latarpratatioaal Rationala 

Aaaaaaaaat of tha "oaaning*' or *'aignificanca'* of validity coaf ficianta 
obtain«d-ta firtt-tiaa validity atudiaa in aaall dapartmantal aanplaa obviously 
ahould be undartakan cautiously.* Infarancas ragarding tha ralativa ioportanca 
or validity of GRE-V, CRE-Q, GRE-Advancad, UGPA and/or othar pradictora,. 
ahould ba drawn only taatativaly and antartaiaad aa hypothasaa calling for 
furthar varification. Attantion should ha focuaaad on discarnibla trands/ 
conalstanciaa, and inconsittancias in tha data rathar than on spacific datail. 

In tha last analysia, questions regarding tha validity of GRE tests should 
be thought of and treated as recurring questions to which up-to-date answers 
should be TOughc frequently and locally, ihaaa first-year studies should ba 
replicated, additional criteria might wall ba examined profitably, and studies 
Involving longer-term criteria are important to the establishment of an informed 
basis for interpreting GRE scores in graduate school admissions contexts. 

The findings of this GRE Cooperative Study will ba of greatest valUa If 
perceived as first approxinatJons in an iterative GRE validity-study process 
that is essentially open-ended* 



♦Tests of statistical significance have not been stressed in evaluating the 
validity coefficients obtained in these preliminary studies. A positive 
relationship between academic predictora and academic criteria is expected a 
priori, hence the null hypothesis (i.e., that no correlation exists between the 
predictor and criterion variables under consideration) is not deemed to be 
appropriate. Following completion of the Individual institutional-departmental 
analyses and reports, suonary distributions of obtained validity coefficients 
will be prepared in order to assess the range of observed values. 



OMcrlpeloa of th« SampUs, Data, and Procaduras 

Tha praaant atudy la baaad on dau for 25 aeudanta In Gaology/Gaophyalea 
and 22 Botany /Microbiology atudanta In tha Unlvaralty Graduata 
School. Both dapartaanu of far doctoral prograoa, and tha aampla atudlad 
Indudaa both proapactlva «aatar*a- and proapactlva doctoral-dagraa candldataa 
for vho« flrat-yaar prograna and avaluatlen procaduraa raportadly ara cooparabla. 
Data wara auppllad for flrat-tlaa anrollad, full-tlaa, dagraa*aaaklng atudanta 
who antarad In fall 1974 and fall 1975, ratpactlvaly. Tha atudy focuaaaa on 
tha ralatlonahlp batwaan GRE acoraa and othar waaauraa of parsonal and back- 
ground charactarlatlcs of atudanta (potantlal pradlctora of parforaoinca) and 
ntaauraa of thalr parforaanca (crltarla of "auccaaa") during tha firat yaar of 
graduata atud!$r. 

Enuaaratad and daacrlbad brlafly balow ara tha varlablaa for which obaar- 
vatlona wara raportad and tha nunbar of atudanta with obaarvatlona on aach 
varlabla: 

Adttlaalona varlablaa Gaology/Gaaphy Botany /Microbiol 

" (n) (n) 

GRE-Varbal ((GRE-V) 200-900]" 24 10 

GRE-Quantltatlva ((GRE-Q) 200-900] 24 10 

GRE-Advancad Taat (200-990) Caology (23) n.a. 
Ondargraduata GPA (UGPA) 25 
(A - A» 3 - B, 2 • C, 1 - D, F - 0] 

Criterion (Performance) variables 

Graciuata GPA^ General (GGPA), Year 1 
[scale as for UGPA] 25 22 

Ad hoc Rating— potential for advanced 
study in the field. Average of 
faculty ratings on a four-point 

scale (see Table 3). 25 n.a. 

Other variables 

Birth Year (last two digits) 25 n.a. 

BA/BS Year (last two digits) o 
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^It Is assumed chat all such students were iocluded on the data rosters 
provided. 
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Informatlon rtgardlng «thnlclty was supplltd fdr the" Geology tanple 
(which Included only two non-Caiicaiian students, both of whoa vara classified 
as "Oriental or Aslan-Aaerltfan") but not for the Botany sample. TWenty of the 
25 Geology students and 17 of 21 Botany students for whom sex coding was 
provided were male. 

All members of the entering cohorts being studied were classified as. 
cpntlnulng In the department. In progress toward a' degree, as of the beginning 
of the second year following admission. Indicating no first-year attrition 
(assuming that all members of the entering cohort are represented In the 
respective, samples). . * 

For Geology sample d^ta were provided on year of birth, year of bachelor's 
degree, and continuity /discontinuity with respect to major field and Institu- 
tion from undergraduate to graduate school. < 

22 of 25 students earned the bachelor's degree In 
1974 (11) or 1975 (11); of the remaining three 
students, two were 1972 graduates and one graduated 
In 1968, 

In terms of age (year of birth) the sample was 

distributed over a somewhat wider range (approximately 

20 to 29 years of age at time of entry Into the. department) . 

• Of the 25 students, five students were graduates ofvthe 
University (continuity of Institution) and 19 had an 
undergraduate major In Geology (continuity of field). 

Limitations of the data 

Apart from the small size of the samples under consideration, observations 
were not available for all Individuals In each sample; GRE Aptitude Test scores 
were not available for 12 of the 22 Individuals In the Botany /Microbiology 
sample. In small samples In which a predictor Is not available for the entire 
entering group, evaluation of the validity of that predictor (and others> is 
doubly compltcated. It cannot be assumed, for example, that the Individuals for 
whom a predictor is available are "like" those for whom it is not available in 
terms of performance on other potential predictors. Individuals with "marginal" 
Undergraduate GP\, for example, may be required or may elect to supplement 
their admissions application with GRE Advanced or Aptitude Test scores; those 



with "very low** GRE-Verbal scores nay have been admitted; In part, on the' 
basis of "compensatorily high" GREoQuantltative scores, etc* 

Even' when all predictors are available for all individuals, there may 
be systematic effects oil the observed validity coeffiodents as a consequence 
of the way in which the variables were employed in screening candidates* 

• The Graduate GPA, reflecting grades received for work completed* during 
the first year of graduate study, is distributed over a very limited range. 
In Geology, for example, only onie student received a GPA below 3.00 on a 
4-point scale: only two c. 22 Botany/Microbiology students earned less than a 
3.00r^<PA} (and GRE Aptitude scores are not available for one of these "low 
performing" students). 



In view of the foregoing limitations, the findings reported herein 
clearly should be thought of as "first approximations" in a continuing validity 
assessment process. 



Due to the missing dara pattern, descriptive statistics and validity 
coefficients were determined by using all observations available for a given 
variable or pair of variables. Thus, for example, the GRE-Aptitude scores for 
Geology are based on 24 cases, mean Advanced Geology score is based on only 23 
cases, and the validity coefficients for these scores are based on 24 and 23 
cases, respectively, having both the GT£ scores and one or more criterion 
scores.. In the Botany sample, GRE Aptitude validity coefficients were determined 
for the 10 students with GRE scores. 



See Robyn M. Dawes. Graduate admission variables and future success. Science, 
1975, 1817, 721-723, for an analysis of tha attenuating effect on predictive 
validity of compensatory methods of screening applications for admission— i.e. , 
selection using multiple assessment variables in such a way that If the 
selected individuals are low on any particular variable, they will tend to be 
compensatorlly high on others. 



Procedures 
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' Oiaracttristlcs of the Samples ^' 

. * * • 

Tabl< 3 ahows naasurat of central tendency (the arithnetic average or 
Man) aiid variability (the standard deviation) for the dlatributions ot 
predictor and perf^man^e (oC* criterio|t) variables in the 'respective saoples* 

* Both departmental samples are characterized by somewhat > 
higher GRE»Q than GRE-V means; the average individual in 

^ these samples ranks slightly' below the 60th percentile on 

GRE<-Verbal, while- the average GRE-Q scorei rank somewhat lower 
than the 75 th percentile nati||^lly» 

* • * 
Standard deviations bf GRE-'V scores are comparable for the two 

samples; however, with respect to quantitative ability, i:he 

Geology/Geophysics sample is more homogenous than the Botany/ 

Microbiology sample, judging from the smaller standard 

deviation of GRE->Q scores in Geology. It is Important to note, 

however, that GRJS Aptitude scores are not available for more 

than half the Botany sample. ^ 

* With respect to scores on the GRE Advanced Test (Geology) , the 
Geology saiiq>le is quite homogeneous (SD • 63) around a mean 
value that is about average for candidates who take this test 
nationally. 

It was noted earlier than the Graduate GPA (first-year work) was distributed 
over a limited range— only one Geology and two Botany students received a 
GGPA of less than 3.00 on a four-point scale. The mean values of GGPA 
(3.47 and 3.45) reflect the dearth of "lower" grades In the respective depart- 
mental samples. Judging from standard deviations, the GGPA distribution for 
Botany is somewhat less homogeneous than that for Geology. 

Variables on which observations are available for Geology but not Botany 
include the GRE Advanced Test, considered above, the Undergraduate GPA, Ad 
hoc faculty ratings* Birth Year and Year of Bachelor's Degree. 

The Graduate GPA, of course, reflects more or less "routine" patterns of 
evaluation and grades in Geology appear to Involve primarily "A's and B's," 
(i.e., 4*8 and 3's on a four-point scale). For the Geology sample, ratings of 
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TabU 3 , 



Mtasurfts of C«nfer*l landtncy *nd Variability 



Cftoloay /Gaophys tea 

Standard 
Mean deviation 



Botany /Microbiology ^ 
Standard 
N Mean deviation 



24 


522 


105 


24 


606 


81 


23 


559 


63 


25 


2.99 


0.24 


25 


3.47 


0.35 


25 


2.49 


0.51 


25 


51.20 


5.04 


25 


74.04 


1.48 



10 
10 



22 



536 
591 



3.45 



101 
100 



0.55 



GRE-Verbal- 
GRE-Quantltative 
GRE-Advanccd (Geology) 

Undergraduate CPA (UGPA) 
Graduate CPA (GGPA) ^ 
Ad hoc rating (average) 

Birth year 
BA/BS year 



*Rated in terms of potertial for advanced study on a four-point scale: 

Outstanding performer, definitely qualified fo^do^^^tM'^^H^i.or.l study- 
3 n Definitely master's caliber; probably capable of acceptable <io"«"^ ""J^ 

: AdeJCaJe ?oTdequate plus at master's level; would «e^--"^ter'S^^^^^^ 
1 . Unacceptable or marginally acceptable for graduate study at master s levai. 

✓ 

Table 4 

Correlation of Predictors witl^ Criterion Variables 



Botany /Mic rob io logy 



Variable 



CRE-Verbal 
GFE-Quantltntive 
GRE-Advancod (Geology) 

(V + Q)/2 or ORE Apti- 
tude Average 
Undergraduate CPA 
[Graduate CPA] 

Birth year 
BA/BS year 





GGPA 


Ad hoc 
rating 


M 


-GGPA 


24 


f 

.13 


.43 


10 


.18 


24 


-.03 


.26 


10 


.42 


23 


.14 


.40 






24 


.06 


.37 


10 


.34 


25 


.27 


-.34 






1251 


[1.001 


[.331 






25 


-.25 


.20 






25 


-.33 


-.07 
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potential for advanced study were provided on a four-point scale* Given the 
new frame of reference reflected in the ad hoc ratings (§ee note to Table 3), 
faculty Judgments of student performance were distributed over a wider range 
(standard deviation of *S1 as compared to *2A for GGPA). 

0 

Correlation of. Predictors with Criterion Variables 

Table A shows correlation coefficients indicating the relationship of the 
respective admissions variables to Graduate GPA and Ad hoc ratings, respecrively. 
In the Geology sample. Compvable coefficients for GRE-V and GRE-Q, involving 
only 10 cases with these scores, are shown for Botany /Microbiology, along with 
a coefficient for a variable called "GRE Aptitude available^C coded "1" if 
scores were present!). In both saaq>les, a coefficient is reported for 
GRK(V + Q)/2 (the average of an individual's V and Q scores, the operational 
equivalent. of the GRE total score more frequently encount)ered in practice). 

Findings for Botany 

For 10^ students with GRE Aptitude scores, GRE-Quantitative and GRE-Verbal | 
are both positively related to Graduate GPA, with GRE-Q. somewhat more closely 
associated (r - .42) than GRE-V (r • .18), The positive coefficient (r • .21)* 
for "ORB Aptitude Available" indicates a tendency for Graduate GP^ to be higher^ 
for individuals with GRE Aptitude scores than for individuals withjout GRE • ' 
Aptitude scores. The nature of the relationship is shown below: * 



Graduate GPA 



ORE Availability Below 3.50 3.50 - 3.74 3.75 plus 



GRE scores available (10) F" T T 

GRE scores not available (12) 6 . , 5 1 

Total 8/7 7 



No description w.is •provided of the .procedures employed in developing ad hoc 
r.itlni?.- In absence hf ex'plicit fnfortnation about procedures, it is 
issuraed that , the ratings were made in accordance with the standard schedule 
for rating students in ternus of potential for advanced study in a fierld. 



Flfi4ingt for G«ology ^ • . ♦ 

The validity coefficients for GRE predictors in Geology suggest (a) tha.t 
Faculty .Ratings tend to be more closely associated with GRE scores than is 
the General CPA.. and (b) that GRE Advanced Geology Test and GRE-Verbal scores 
may tend to be more closely associate<< with pertormance criteria among enrolled 
students than GRE-Quantitative. ^ 

• Coefficients for GRE-Verbal are positive with retipe^ct to both ^ 
GOPA and Ad hoc ratings, but somewhat larger when Ad hoc ratings 

(r « .43) constitute .the criterion than when GGPa is the 
criterion (r - .13). GRE Advanced Geology has a comparable 
pattern of validities *(r • .40 vs. Ad hoc ratings; r - .14 vs. 
GGPA). A small negative coefficient was obtained for GRE-Q vs. 
» GGPA (r - - .03); when Ad hoc ratings constituted the criterion, 
the coefficient for GRE-Q'was positive (r • .26). 

• The. Under graduate CPA is positively associated with Graduate GPAj 
but negatively associated with Ad hoc ratings in this sample. ^ 

« 

This finding must be considereil anomalous. The expectation, a priori, 
is that^past academic performance will' tend to be positively 
* associated with future academic performance and that different 

measures of performance will tend to bft positively associated. 

The two criterion (petformance) measures are positively associated although 

the relatiwi^lp is relatively low (r - .33, Table 4). Ungthy speculation 

about •♦reasons*^' for findings of this tyi^e is not warranted here. However, it is 

useful to note that in small samples one or two "aberrational" data sets can 

have considerable influence on outcomes. In the present Geology sample, for 

example, two Individuals had unusually high Undergraduate CPA (3.65 and 3.61, 

compared to the mean value of 2.99, standard deviation of .24; only two other 

UGPA greater than 3.00)— both of these individuals were among the four lowest 

ranking studoncs In- terms 'of Ad hoc ratings and, thus, contributed substantial , 

nej5aclve co variance in the UGPA/ Ad hoc rating analysis. 



*Bv cx.iralning the original data rosters, the Department can identify the two cases 

involved. It would be of some Interest to look Into the records of the students 
involved in an effort to Identity circumstances that might help to "explain" the 
except lonally high undergradilate record and the "very low" Ad hoc ratings. The 
possibility of clerical error should be considered In such an examination. Perhaps 
the undergraduate institutions had somewhat /"lower standards," etc. 

r ' 



No « priori Mt of oypoctatlons can bo odducod te guldo ovaXuoelont of the 
pattom of cqtfficionts for Birth Yoar (Inversely related to age) and Year of 
Bachelor's Degree. They are shown only as descriptive of tendencies within 
the .j^resent sanple. In this sense, ylth respect to General Graduate CPA, the 
negative coeifficlent (r • -.25)^ for Birth Year Indlcdtes a tendency for 
younger students to, earn lower CPA, and the positive coefficient (r - .20). 
with respect to Ad hoc ratings Indicates the opposite tendency. In the case 
of Year of Bachelor's Degree, with respect to both criterion variables there 
Is a tendency In the sample for more recent graduates to receive lower 
criterion. scores. However, as suggested above, there Is no a priori rationale 
for evaluating these findings and they should not be generalized. 

Valldl«y.u>f Composites 

In view of the small size of these samples and the anomalous patterns of 
coefficients there was no application of formal multivariate procedures for 
establish^ the relationship of two or more predictors to the respective 
criterion variables. However. It is of interest to call attention tc the 
relationship of GRE(V + Q)/2 to the criterion variables (cf . . pp. 7«8, for 
discussion of weighting). It may be noted (Table t^) that this simple average of 
GRE-Verbal and GRE-Quantltatlve yields lower validity coefficients with 
criterion variables than GRE~V or GRE-Q, whichever is higher. When CRE-V 
and GRE-Q, as in the present samples, have quite different individual 
validities with respect to a c .terlon variable, the GRE Total (or GRE Average) 
will tend to have less validity than one of its components— the sum thus has 
less validity than one of 'its parts. The point to be made here is that 
composites of GRE scores that jjave apparent "face validity" (i.e.. appear to 
be loRical) may not in fact be appropriate. Empirical validation is needed for 
all GRE composites. 

In the present exploratory study, questions of relative weighting for 
GRE-V. GRE-Q. (and UGPA in the case cf Geology) cannot be addressed directly. 
It Is. however, of interest to call attention to the possible impact of 
restriction of range within departmental samples on the patterns of validity 
coefficients for predictors. Ja^the present situation, for example, GRE-Q has 
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lowttr eoTMlatlon vlth Criterion variables than GRE in the Geology aanpU and 
hlghtr correlation than GRE-V in the Botany sample* Questions of sampling 
fluctuation aside, it is relevant ^o note thae the standard deviation for 
GRE^ is considerably smaller than that for 6RE-V in the Geology sample whereas 
comparable standard deviations obtain for these two predictors in the Botany 
sample. The Geology sample is more restricted with respect to spread of tJRE-Q 
than with respect to GRE-Verbal scores, a condition which tends to be 
associated with somewhat lower validities for the variabla characterised by 
restriction. 

Cone lu8 ions and RecomMndatlons 

As indicated at the outset, this study should be thought of as yielding 
first approximations in an iterative GRE-Validation process that is essentially 
open*>ended. The findings suggest hat GRE scores contain information of value 
when considered along with other information used in screening applicants for 
admission. An anoi^alously negative validity for Undergraduate GPA with respect 
to one of the criterion variables should not be thought of as providing a basis 
f?>r generalization— such a coefficient in a small sample may be influenced by one 
or ^ two aberrational data-sets* Attention has been directed to two such data- 
sets in the Geology sample. 

The need for replication of validity studies is clearly indicated. Possible 
directions for further examination of the validity of admissions variables 
(ORE and others) are suggested below: 

• The highly restricted nature of the Graduate GPA distributions, 
involved has been noted. The fact that grades tend to be 

« 

distributed primarily over a one-point range in a four-point 
grade scale tends to limit the utility of this variable as a 
criterion. Faculty ratings appear to be a more predictable 
criterion than Graduate GPA. Results of this study suggest the 
Importance of further exploration of the utility of systematic 
faculty ratings of students in terms other than those specified 
by routine grades-in- course. 
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A variety of racing procedures may be thought of as^ being 
"adtelnlstratlveJy viable" and such procedures. would permit 
assessment of student performance (if only for pu^oses of 
research and evaluation) outside the traditional GP A- system that is 
conceptually more applicable to undergraduate than to graduate 
settings* 

Replication of one-year validity studies such as those reported 
herein would be facilitated if rosters of first-time enrolled 
students enrolling each year were developed by each department. 
vJRE scores, UGPA, and other potentially important admissions data 
for each such student could be recorded. These rosters could be 
updated with General Graduate GPA and other performance data, 
including systematic faculty ratings, at the end of the first 
year of graduate study (£.nd subsequent years of study, as well). 



Kenneth M. Wilson 
June 19 Z7 
Princeton, NJ 
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Appendix C-2 

ORE COOPERATIVE VALIOITT STUDIES PROJECT 

Educational Testing Sarvlctt 
Prlncaton', MJ 08541 

To: Participating Graduate Schools " 

Date: March 1<»78 

Fron: Kennath M. Wilson 



Subjact: tabular Suaimary of Salectad Validity Study Findings 



Attached Is a taoular summary of selected findings of studies that have 
been completed as part of the GRE Cooperative Validity Studies Project, 
sponsored by the Graduate Record Examipations Board. The Cooperative 
Studies have been concerned with the relationship of GRE Test scores and/or 
Undergraduate Grade Point Average (UGPA), as available, to First-Year 
Graduate Grade Point Average, as well as other criteria of performance 
during the first year of graduate study. To date, validity study reports 
have been prepared for 34 graduate schools; for several other schools data 
are currently being analyzed and it is expected that a faw.more schools 
will provide data (see list). The data in this suoaary reflect selected 
findings of the 34 completed studies. The summary is intended to provide 
general normative perspective to facilitate assessmant and evaluation ol 
trends and- patterns across graduate schools and departnants» 

The "common criterion" in the Studies that have been completed has been the 
first-year Graduate GPA (GPA General), Other criteria such as ratings 
or examination scores are very unevenly available. Table A summarises 
infotmatlon regarding the number of samples for which data have been 
analyzed by discipline or department. The number of samples varies con- 
siderably across disciplines. This table also shows weigh red ^an values 
o£*validlty coefficients for GRE-Verbal (GRE-V), GRE Quantitative (GRB-Q), 
GR2-Advanced (GRE-Adv) , and Undergraduate GPA (UGPA) , as available. 

The weighted mean values reflect degree of within-group covariation between 
relative standing on the respective predictors and relative standing on the 
Graduate GPA criterion in pooled samples of all Individuals with predictor 
and criterion scores. The coefficients indicate what the predictor- 
criterion relationship would be in such pooled samples after all predictor 
and criterion scores had first been standardiz ed within each of the samples 
Involved- -le», standardized within each departmental sample and then pooled 
for analysis. 

In one or two instances a Critical Graduate GPA (e.g., grades in required 
or common sequences only) or an average of two or more criterion variables 
is involved, rather than the Graduate GPA General. Some clustering of 
f leld/departaents nas been Introduced, as indicated In notes to Table A. 
The data in Table A are of interest from several points of view: 
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l» Numbers in parentheses following the number of departmental samples 
involved Indicate the number of samples vich Ns for validity coeffi- 
cients that were equal to or greater than the number suggested as 
the minimum target for the Cooperative Studies, namely, N • 25, 
which by design was to be reached by combining data for two succes- 
sive entering cohorts of first-time enrolled, full-time, degree- 
seelcing students. In the aggregate, 86 of 137 Aptitude coefficients 
(about 63 percent), 56 of 81 UGPA coefficients (about 69 percent), 
and 27 of 69 Advanced Test coefficients (about 39 .percent) were 
based on 25 or more cases* The remainder were based on fewer than 
25 cases* 

2. The pattern of coefficients suggests the potential importance of the 
predictor that frequently was "least available", namely, GRE Ad- 
vanced Tests* Advanced Test scores tended to be missing for a 
number of individuals in the samples under consideration* The 
influence of availability versus unavailability of the GRE Advanced 
Test score (or scores on other predictors) on the observed patterns 
of vf.lldity coefficients cannot be estimated* In a number of 
instances, more than one Advanced Test (field) was represented in a 
data-sample e*g*, a " chemistry sample may have Included not only 
students witn Chemistry scores but also one or more with Mathema- 
tics, Physics or Engineering* Despite missing data limita- 
tions and occasional Advanced Test-field heterogeneity, the 
weighted mean coefficients for the Advanced Teats (which 
reflect aptitude, motivation, and substantive achievement) 
suggest their potential Importance* 

3* Generally speaking, the average values of the coefficients in 
Table A clearly are consistent with the working proposition that 
a positive association exists between measures of developed ability 
and acnlevement (such as GRE Tests and UGPA) and measures of first 
year performance In graduate study, such as the Graduate GPA 
general* 

In evaluating the magnitudes of the coefficients, it Is important to 
recognize that the Graduate GPA criterion used frequently (typically) had a 
severely restricted range, and was sometimes simply dlchotomous and heavily 
weighted In the direction of "passing" marks** In the circumstances, the 
size of tne pooled wlthln-group average values of validity coefficients 
undoubtedly are lower than would be the case If differences in student 
performance w«»re mote rigorously and reliably rnflected in the "routine" 
grading process* 



**»hen dlchotora^u.^ GPA criteria were used, point biserial coefficients were 
omputed. Since the underlying criterion variable Involved was actually 
continuous, the point blscrlal underestimates the relationship of the 
continuously distributed predictor to the underlying continuous criterion 
variable. 
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SumoftrlsAClon by Field and School * 

Trie data in Table A provUa tvldance cegatdlng tha typical lavala and 
patteens of coefficients by field.- Data luminarUed In Tables I through U 
show, for designated fields or groups of fields /depart aents (a) neant and 
stdndard deviations of predictor and criterion scores In aach sample, (b) 
Intormatlon regarding sample »lae, and (c) the obaerved validity coeffi- 
cients. 

♦ * 

Validity coefficients are not- reported If based on leas than H • 10. In 
such cases, the sign of the observed coefficient .i« reported to indicate 
the direction of the relationship in the sapple. It la Important eo note, 
however, that (a) the typical level and pattern of coefficients in these 
very small samples followed closely that for all samples as reported In 
Table A. and (b) that the weighted coefficients derived from these ex- 
tremely small .samples were Included in developing the weighted mean values 
reported in. Table A. 

Host Graduate CPA am Undergraduate CPA scales were A - 4, B - 1, C • 2, 
etc.; occasionally the GPAs reported by schools were on other scales. For 
uucposes of the summary presentation in Tebles I through U, means and 
standard deviations based on atypical scales have been adjusted to make 

ch<im roughly comparable. « i 

£n some Instances, multiple correlation coefficients. based on missing data 
correlation procedures are shown (in parentheses under the UGPA Column). 
Unless otherwise indicated these coefficients are based on all 'the predic- 
tors for which validity coefficients are reported. Multiples are not 
routinely reported due to sample-sUe, missing data and other related 
considerations. The purpose in reporting multiple correlations in these 
saall samples is primarily to provide some perspective on the potential 
value of combining two or more predictors* 

Several features of the data in Tables I through II are noteworthy, includ- 
ing the following: 

U Despite the limitations of missing data procedures, the multiple 
correlation coefficients suggest that the test variablee and the 
undergraduate grade point record tend to provide some uniquely 
important information about student performance-potential. 

I, In examining the tables It will be seen that "useful" l«vel» 
wlthln-group validity are to be found in data for samples differing 
considerably in level of scores on the GR£ predictors. 

J. The interdepartmental data provide useful normative perspective 
regarding the range and patterning of validity coefficients (In- 
cluding occasional negatlvt' coefficients). The potential value of 
the GRE-Advanced Tests, suggested by the average values reported in 
Fable A. is also suggested in the individual departmental analyses 
where coefficients for the other available predictors may be seen. 
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It U important to k«ep in odnd that the eoftfficitnts in Tablet I. through 
II have relatively large aampling errors due to the snail sise of the 
respective saaples. Conclusions regarding the relative validity in partl-> 
cular samples o£ V, Q, and Advanced Tests, and UGPA call for the accumula- 
tion of a substantial body of empirical evidence derivable only through 
replication of validity studies. 
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a 

Graduate Schools Participating in 
. Cooperative Studies 



School 

~- — ~~ ' 

Air Force Institute of Technology 
Auburn .University 
Baylor University 
Bradley University 
Brown University 

California State University at Fuller ton ♦ 
Florida. Technological University 
'Fort Hays Kansas State College 
Harvard University 
Hofstra University 
Indiana University at Bloomington 
Louisiana State University 
Loyola University at Chicago 
The Ohio State University 
Old* Dominion University 
Oregon State University 
Princeton University 
Stanford University 

State University of New York at Stony Brook 

University of Arizona at Tucson 

University of California at Berkeley 

University Of Colorado .at Boulder 

University of Hawaii at Honolulu 

University of Illinois 

University of Kentucky 

University of Massachusetts 

University of Miami 

University of Michigan 

University of Missouri at Rolla 

University of Montana 

University of New Orleans 

University <of North Carolina at Chapel Hill 

University of Notre Dame 

University of Oklahomia 

University of Texas at Arlington 

University of Virginia 

University of Wisconsin 

Virginia State College 

Washington State University at Pullman 

Wayne State University 
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• 


Seheel lit 
(13,13,13,13.) 


413 
61 


'675 
64 

• 


3U 
97 


2.94 
0.43 




Seheel :04 
(32,32,9,—) 


43» 
UO 


661 
102 


709 
100 




N 


School 2U 


336 
107 


679. 
63 




.2.^2/ 
0.39 


M 

s 


School 24« 
(M,M,— .30) 


307 
64 . 


' 361 , 
7« 




2.36 
0.33 


M&O 


School 293 
(I9.l».— ) 


283 
66 


632 
66 






M&O 


School 293 
(14»U*— ) 


433 
101 


649 
67 







Tis Ztt (-)* -.u 



OunHLcAl CatiiaM<ittS dot*) 
(♦)• (♦)• - 

OMalcal tatiseerlsg (aonfer.) 
.J4 .a - — 



lasiaeeriat HaMfwndiit (for.7 
«26 .4t .74 (4>)* 



.12 -.30 -.03 .41 



tlectrieal laclaeerlat 

,24 ,42 (♦)* — 



Uee«rie«l lagiaeeviai 

,46 ,34 — .61 

• ^ . 

r«ciltti«s Ma^eaeae 

TSr .at — -,23 



Civil tatlaeerisg (fer.) 
.U .30 — — 

CivU EntifM«rln| tneafet.) 
.30 ^,47 — — 



3.35 
0,31 



3,40 
O.U 

3,49 
0.34 



3,60 
0.20 

3,62 
0,26 



3,41 
0.27 



3,31 
0.3S 

3.72 
0.16 



3,46 ' 
Q,41 

3.37 
0.32 



Nettt Nulttolt eor^tDtlon eoeffleitnt; (K valutt in D«rtntnt««« 
In UGPA column) art not rtoertt^ln Hi instances due ori- 
eerHy to saeoie site and/or eissinQ data considerations* 

•Ceefflclenc b«s«d ea less ch«a 10 eeees; site ladleaeee dlrteelon 
of relaeloaship, 

^Indlca^es degree orlentaclon of students In sample: M • oaster^s, M&O 
both prospective nastfer ' s. and doctoral students. 
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V 



tummttY of Validity Cofffldtntt for QRC -anO*^ < 
ua#4 PPo4i<tor« voMut F{rtt*Yo«r 8r«4u«tt 

MATHtMXTtCS. COM?qTg> SCmCt 4 ICOWOMICS 





School (coa«4) 


rrodlceWr moan 4 a 






CI / coofeicione) 












School 101 
(U,U.12.14) 


S74 
74 


744 
IS 


741 
90 


3.71 
0.30 




School 132 
(43.43,22.—) 


810 
98 


704 
78 


(673)* 


— 


H 

• 


School 169 
(U.U.— .18) 


599 
108 


687 
83 


- 


3.25 
0.45 


M&O 


School 204 


338 

115 


703 
86 




— 


M&O 


School 204 


488 

123 


881 

102 




— 


M&O 


School 204 


347 
137 


691 
94 






D 


Scnool.033 
()7,)7;Z6.— ) 


880 

97 


712 
IP2 


790 
93 




M&O 


School 097 «• 


• 56 
26 


•75 
21 


65 
23 


3.53 
0.35 


M&O 


School 118 
(20.20.— .r-) 

School >4S 
(36.36.— .37) 


508 
99 


610 
76 






M&D. 


597 
74 


700 
65" 




3.54 
0.31 


M&D 


School 204 - 
vW.l»\~.— ) 


473 
136 


535 
121 






M&O 


Scnool 320 
(^9.^9.41,50) 


587 
102 


672 
85 


645 
89 


3.16 
0.54 



CtU CfA 
M4n 6 sTo. 



ItotlMMCUt 
.38 -.Ol 



.58 



Choa/KMhoaotlet 

.55 .40 — 



MftthoMCles 
.24 .40 — 

Ap»Ito4 Moth 
.27 .27 — 

Coaputor ^cloMo 
.34 .31 — 



Eceneaict 
-.14 .01 

Ceoaoaic* 

.05 • .29 

EeeiuMULet 
.06 .27 

Eeonoalet 

.17 .37 

Econoaloo 

.37 .56 

Ccenoaies 

.14 .55 



.32 



Coa^et* Seioaoo 

.25 .08 .22 — 



.29 



.49 — 
.53 .18 



.02 
(R- .38) 



.33 .53 
,71) 



3.03 
0.41 

3. .64 
0.55 

■ 3.39 
0.47 



3.37 
0.53 

3.32 
0.49 

3.79 
0.41 



3.08 
0.59 

3.60 
0.38 

3.*,6 
0.26. 

3.15 
0.40 

2*91 
1.00 

3.46 
0.J6 



Nott: MuCtiolt coftUtion cotff'cltnts (R v4lut« o*fontr»t«t.» 

in [iGP^ coCumo) *p» not rtoopttd In all <n»t*nc»$ dut 3p<- 
•arily- to »*«ioCt %if *nd/0P ulttlnfl data conri dtrat ion*. 

*Etsh6 41ff«r«nc A4v«nc«d loot floldo. lorgoiy .H«th«a«tlc«; no , 
Coapwtor Sclooco Scoroa. 

**P«ccoacllft rooks 

'^Indicates degree orientation of students in sample: 0 • prospective doctorol 
students only. M&O - both prospective awster's and doctoral students, M - 



prospective master's only. 



ERIC 



u«««fy •* V«ll41ty C«*ff1c1«nts for QRC «n4 

msies . CEOtocT. ctopgrsics 



School <co4«4) 



frtdlctor wan t o.4« 





(S / ce«(flcl«ie) 










D 




lot 


74» 
67 


815 
111 




D 


School 03S 
<16.1*.15»-) 


•7 


746 
SI 


685 
89 




0 


School 035 

/lA tft < __\ 


«13 
110 


743 
44 


728 
58 




M&D 


School 04« 
(41»41»40»43) 


5U 
99 


720 
69 


744 
97 


3.52 
0.28 


M&O 


School 04« 
(39»39»32»39) 


373 
105 


642 
76 


. 660 
82 


3.21 
0.36 


M&O 


School 097 
(9I»98»99»10S) 


417 
91 


749 
53 


743 
99 


3.60 
0.31 


0 


School tOl 
<9»9.9.— X 


536 
lU 


696 
79 


579 
97 




M&O 


School 249 
(24»24»23. 24) 


522 
105 


606 
61 


359 
63 


2.99 
0.24 


M 


School 293 
(9»9»— »- ) 


472 
62 


604 
71 






M 


School 293 


579 


715 
37 







CXC-V Qwq CHl^My urcA 



Gtii C?A 
«•«> 6 t.6. 



rhyoiMu 
.06 •.04 

Goolety 
-.01 -.06 

Coophjroles 
.01 .32 



Physlet . 
.02 .16 

Coelogx 
.09 »14 



-.01 f .17 

Phyolca 
(♦)• (♦) 



.10 
.06 

4 

(-)• 

.31 
.18 

.11 



.23 
(&• .40) 

.37 
(a- .37) 

.31 



6«elosy/C«ophyolc« 
U .03 .14 

Geology 
(-)• (-)• — 
fhyoico 



.27 



3.36 
0.68 

3.72 
0.18 

3.83 
0.39 



3.57 
0..40 

3.51 
0.26 



3.44 
0.40 

3.26 
0.57 

3.47 
0.35 



3.80 
0.31 

3.57 
0.37 



Noto; Multiplt coi*i*tl4tioii cotfficltntf <« V4iut« in otrtntntttt 
in celunn) art not rteorttd in au instances Cut sri- 

laariiy to sa*elt s'itt and/or aiissing data considtcat ions. 

*Co«fflcl«oe ba««d es l«aa eb«A 10 c«»««; sign lodicacos dtracclon 
of roladonohip. 

Indicates degree orientation of students In sample: U « prospective doctoral 
candidaCes, M&O • both prospective nuister's and doctoral students^ M • master's 
only. 



-151- 



V 



SwMaary of Validity CQ*ff1«iti *s for SKI and 
UQ^A fftdictort vtrsut f1rtfY««r 6r«du«tt 

CKCLXSA. SrCSCH. CO»MC!IXCATXO!I . TRCA7EX 



SctMol (codod) 
(9 / coof eieloAC) 



fg^dleeor «oan > <.d. Valtdity eooffteioiie Crad GPA 

Cai-V CU-^ Oli-Adv UCTA dftk-V Ctt^ Cll-Adv MM • C.d. 



if 

M/D 

mo 

M&D 
MdD 
M&D 

M 

H»- 

M 

M&D 
M 



School 01S 
<2l.2l.l$.— ) 


499 
«3 


iS 


824 
97 




School 123 
(24.24.17.23) 


381 

lie 


308 
98 


339 
79 


3.28 
0,50 


School 132 
(19.19.14.23) 


848 
81 


316 
109 


833 
81 


3.49 
0.26 


SCStOOi i**) 
(4d.4«.— .40) 


8d 


101 




0.32 


(28.28.20.—) 


390 
129 


494 
130 


381 
99 




School ::i 

(32.32.^8. S2) 


652 
88 


372 
102 


391 

77 ■ 


3,39 
0.27 


School 007 
(-O.iO.— ,40) 


492 
88 


473 
102 




2.U 
0.41 


School 122 
(31.30.— 32) 


477 
U7 


438 
Ul 




2.91 
0.36 


Scnool 123 
U6,3ft.— ,38) 


322 
90 


477 
99 




3.07 
0.31 


School 123 
(38.38,— ,—) 


478 
118 


433 
78 






School l-»S 
i3«,39,— ,39) 


3^7 
97 


309 
108 




3. SO 
0,26 


Scnool •31 
v60,60.— .72) 


364 
109 


490 
U4 




3.08 
0*39 


School 231 
(44 .44.33,— ) 


443 
94 


329 
99 


592 
87 





,S9 .15 

,30 .21 

.22 -.0* 

Sa«ll«h 

.23 .18 

.72 ,4S 

.44 ,34 



.81 * — 

,1% 



.40 



.26 
(K* .38) 

,28 
(»- .31) 

.08 
(f .38) 



.62 

.40 



.29 
(»- .59) 
Sp«tcli «ad C oM untci Cioft 

.41 .42 — .31 

Oram* and Co —un teatiott 



.27 



.23 — 



Sp««ch aad Thaacar 

.34 .23 — 

riaa Area 

.50 .13 — 



Spaaeh aad Thaacar 

.35 .13 — 

Jeuraalism 
.21 .34 — 



.30 
(t - .36) 



,27 
(» - . 36) 



.30 
(R • .43) 

.33 
(R - .43) 



Xt^Axalv 
.22 .15 



.17 



3.76 
0.83 

3.65 
0.35 

3.63 
0.29 

3,62 
0.36 

3.68 
0.29 

3.07 
0.39 

3.54 
0.49 

3.62 
0.36 



3.42 
0.49 

3.56 
0,41 



3.42 
0.3S 

3.37 
0.29 

3.6S 



Not*: NuUloU eorrtlatlon eo*ff<ei*nts (R valuas in oaffnthtsas 
^ in USPA column) art not rtoortto In all Instances out ori- 

marily to sa«olt sitt and/or missing data eonsidtrationa. 

•This coafflclaoc la for a eoapoalea of pradlecora aaad in 
adalaaloaa. ^ 



Indicates degree orientation of students in sample: D « prospective doctoral 
students only, M/D • master's plus some post-master's doctoral students, M • 
\^ master's only, Hf « largely master's plus several doctoral students with 
master's degree. 



ERIC 



i5 



0'< 
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Su«««fy 9f V«li«1ty Co«ff1eUAtt fof em and 
•tSTOIT. kttk STODICS. AVTRtorOlOeY 



M&D 

0 
M 

H 
M 



;-A4v UKA MM * t.d. 





494 


403 


999 






.:tf 


- .31 




74 


. 94 


94 






SelMol 044 










liatocT 






419 


935 


943 




.10 


.20 


.09 


(31,31,13,—) 


47 


94 


40 








School OM 










liseoTT 






601 


541 




3.39 


.14 • 


>.14 




(34,34,— ,31) 


U3 


107 




0.42 








School 097 










Riseotr 






612 


938 


943 


3.94 


.04 


.24 


••10 


(30,30,24,32) 


87 


94 


74 


0.34 








School 403 










liseoT7 






594 


992 


997 


3.42 


.38 


.49 


.15 


(34,34,24,24) 


91 


121 


99 


0.38 






Scaool 123 










tlttorr 






589 


920 


934 


3.14 


.58 


.34 


.72 


(27,27,14,27) 


87 


111 


41 


0.91 









( &• .U) 

( t- .21).. 
.02 

.40 

.43 



.38 
( t- ,49)* 



School 149 64? 546 — 3.59 

(48,48,-. aid) 99 U6 0.39 

School 145 691 574 — 3.32 

(37,37,— ,55) . 78 103 . 0.92 



Bi«eorr 

.20 .20 





School 221 
(43,43,43,32) 


649 
80 


602 
107 


594 
73 


3.62 
0.28 


Hf 


School 231 
(24,26,22,26) 


603 
109 


509 
121 


570 
79 


3.34 
0.50 


M&O 


School 009 
(37.37,-,-) 


642 
90 


582 
85 






M&O 


School 145 
(39.39.-,47) 


649 
118 


580 
109 




3.62 
0.30 



School 204 502 
VI9.19,— ,— ) 143 




— .20 
( »• .29) 

A«l«a Studio* 
.44 .35 — .12 

( a« .68) 

littory 
.29 .22 .34 .33 
( &• .62) 

KUeorr 
.42 .49 .02 .43 
( A- .56) 

Aaehropolofy 
.07 .11 — — 

Aaehropelofy 
.41 .30 — .06 
( &• .46) 

Aathtepolosr 
.31 .20 — — 



3.94 
0.99 

3.81 
0.24 

3.91 
0.39 

3.47 
0.91 

3.64 
0.31 

3,47 
0.91 



3.42 
0.43 

3.33 
0*44 



3.26 
O.U 

3.44 
0.57 



3.62 
0.37 

3.98 
0.40 

3.45 

0.37 



Not*: •^uitiolfl\ eorrtlatton eot'Hdtnts (8 values m oaf«nth*$ts 
in UQPA 'colunn) art not rtoortod in aU instanett out 0ri* 
«arHy to saaplt sitt and/or aissing data eonsidtratlont. 

*Co«f(lel«ae tiaaod on V, Q, «a4 UCFA, oaly. 

^Indicates degree orientation of students in the sample: D > prospective 
doctoral students, M&D • both prospective master's and doctoral students, 
M - master's students, M-f • largely master's plus several doctoral students 
with a master's degree. 
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SuM««fy «f VtUdfty CotffUUntt for (SRI tnd 
POLtTtCAL set. PUIltC AOMtStSTItATtO)! . SOCXOl. UHtAil KAN. SOCIAL WORK 



School (co4«4) Prtdtceor wm * 

(M / co«l(UUne) CICoV CMH) CttHUv UCfA 



teianc 



AdvUfCA 



Crad OA 
A ••4. 



miuui seiiuui 

.44 .44 — .90 

«l • .49) 
Fublie AdaittUtratien 

.92 .24 .50 .04 

01 • .95) 

rolitleai SeUae* 

.99 .24 — . — 

Govt. «ad rotsiga AffAltt 
.59 .94 .49 .la 
a • .59) 

.51 

(♦)• — 
(♦)• (♦)• 



M&D 


School 149 
(90.90«— .99) 


«12 

U7 


604 




9.99 


H 


School 221 


527 


505 


461 


3*16 


M&D 


School 231 
(32»32.— ) 


562 
114 


576 
196 






M&O 


School 320 
(79,79.61,69) 


611 
66 


569 

m 


551 
69 


9.44 
0.99 


M&D 


School 046 
(27,27,26,27) 


59S 
U2 


509 
12S 


550 
97 


3.01 
0.54 


M&D 


School 097 


600 
69 


551 
99 


594 
•9 




M 


School 122 
iS.7,8.8) 


497 


466 
67 


900 
77 


3.00 
0.91 


M 


School 143 
v«*3»43»— •43) 


$97 
96 


625 
125 


mm 


3.41 
0.41 


H&O 


School 145 
(27»27.-.J6) 


601 
LIO 


606 
115 




3.42 
0.47 


M 


School 149 
V27.27.-.2:) 


494 
111 


519 
142 




3.15 
0.40 


M 


School :93 


540 
95 


4S9 
10? 







Socio lotr 

.41 .96 

Soeielenr 
(♦)• (♦)• 

Soeiolocy 



.64 
d • .76) 



rublle. Poller 
.50 .69 

Soeiolotr 
.44 .26 

UrbM PloaalBg 
.27 .29 



Social UOrk 
.43 .14 



> 

(R - .60) 

.47 
a • .65) 

.54 
a • .58) 



9.44 
0.42 

9.10 
0.97 

9.49 
0.96 

9.59 
0.92 



3.44 
0.29 

9.74 
0.20 

9.98 
0.48 



3.23 
0.49 

3.46 
0.40 

9.35 
0.92 



9.42 
0.96 



Not*: MuU^olt corr* l«t < on- co«f f i ei«nts (R valuts in oar^nthtsts 
in UCPA column) art not reported In aU instances due ori'» 
na^ily to saoel* sise and/or nissins data considerations. 
•Coefflelonc baeod on th«o 10 caooa; aiga iadicatoa diroecioa 

o< rolacionahip . 

indicates degree orientation of students in the sample: M&D - both prospective 
aujster's and doctoral students, M - aaster s only* 
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Sw«««fy of ¥«U4lty C0tff1cUnt« foe SKI 



VSYCROLOCT 



Stiwol (c»4«4) ?r^«^tgtor m«.nn > t.d* Vaildtev ee«rftetane Cfad C?A 

V CMff iaim) eU>V «11C-A4v W9A CSt-V aiC-A4v UPGA « 



0* 


S«ho«l 221 
(4S.4$,M,42) 


640 
104 


631 

104 


623 
93 


3.63 
0.28 


,42 


,45 


.38 


.22 
(f .51) 


3.14 
0.5S 




SciM«I US 
(««.St,— ,M) 


663 
96' 


633 
US 


— 


3.62 
0.34 


.35 


.35 




.17 
(f .38) 


3.62 
0.34 


H 


School 122 
(34,34,— ,34.) 


337 
83 


368 
84 


— 


3.03 
0.43 


.20 


.15 




.32 
if .44) 


3.10 
0.71 


H 


School 0S7 
(25,2S,a,-) 


632 
107 


619 
H 


640 
81 


— 


-.08' 


-.0? 


.17 




3.53 
0.39 


M 


School 113 
(47,47,4*,—) 


311 
66 


348 
93 


— 


3«40 
0.40 


.44 


.21 




.52 
( &• .80) 


3.72 
0.25 


K 


School MS 
(27.27,-,-) 


367 

103 


544 
10* 






.18 .21 
(Crludon • 


CTA 6oa f CriUcal) 


3.63 
0.48 


• 


School 093 
(22, 22. 22.2:) 


609 
68 


614 
91 


609 
38 


3.20 
0.41 


.07 


.19 


.17 


.18 

• 


3.62 
0.32 


D 


School 297 
120.20,—,—) 


621 
60 


627 
100 






-.36 


.19 






3.70 
0.24 


M&D 


School 097 
(76,76,76,—) 


621 
63 


648 
93 


639 
72 




•14 


.11 


.25 




3.82 
0.17 


D 


School 069 
(32.32,30,22) 


686 

70 


655 
64 


80S 
80 


3.43 
0.26 


.22 


.48 


.71 


-.01 


3.34 
0.30 


0 


School 132 
(33,33.47,51) 


622 
101 


417 
111 


603 
90 


3.42 
0.40 


.52 


.54 


.48 


.11 
( %• .64) 


3.58 
0.33 


0 


School :03 
<5l.31,Sl,— ) 


398 
72 


381 
84 


604 
39 




-.06 .13 .42 3.58 
(Avoragt o( C?A uA Oopt. IUelnt)0.60 


M/D 


School 123 

<;7.2;.i3.:6) 


348 
il 


364 

* # 


5T6 

79 


3.28 
0.41 


-.16 


-.31 


• «»6» 


.29 


3.39 
0.16 



in UG^A column) art not ftOOfttd in tU inttancts dut Ofi^ 
mMfiif to I4«0lt list tntf/or ^ittln^ dttt contidtrtttont. 

^Indicates degree orientation of students in the sample: M&O » both prospective 
master ^s and doctoral students^ D • doctoral candidates » M « master's candidates » 
? • not indicated by department^ M/D • master's plus some post-master's doctoral 

students* 



S«i«ii«ry «f Validity eotffUitntt for 0*1 •n4 

O^A 

lOBCAtX^H A»0 tXltAttSClMCl 



SeltMl (ca4«4) 
Ql / «o«ff lelMC) 



Cea4 CTA 
« ;4, 



n 


SCRMl 007 
(U,4l»— ,41) 


49 


U 




9 M 

0.37 


M 


School 0«9 


443 
19 


440 
83 


473 
43 


3.04 


M 


School 122 
(70,70,—,-) 




435** 
94 






H&O 


(30,30,— ,4» 


326 
U4 


321 
131 




3.22 
0.47 


H&O 


School 231 , 
«2,<2,32.190) 


327 
122 


319 
' U3 


323 
U4 


3.11 
0.39 


M 


School 297 
(34,3*,-,-) 


430 
67 


473 
90 






M 


School 323 
(27,27,— .27) 


424 
78 


471 
93 




2.78 
0.41 


M&O 


School 332 
(12,12,—,—) 


467 
94 


314 
44 






M 


School 028 
(27,2t,-,2a) 


328 
97 


436 
123 




3.12 
0.39 


M&O 


School 143 
(3l.3l,-,32) 


394 
99 


323 
123 




3.42 
0.42 


M 


School 221 
(29. 3*,-, 38) 


610 
109 


535 
121 




3.28 
0.67 



i 

a9 



.33 



SfMlal Cd 

^^20«* 

Edttcaeiott 
.33 .22 

tdueaeloa 
.14 .24 

rh7«ie«l U 
-.22 ^.29 



.34 

* 



«l • .33)* 

.44 
41 • .40) 



.34 •' 
01 • .39) 

.19 
01 • .57) 



Adalaictraelott «ad Su^orrieloa 
.U .11 — .1* 

^ocatioMl-Toehaioa Sd 
.23 . .23 — — 



LibracT S€l«ae« 
-.14 .30 — 

UbrMT Seloae* 
.44 .59 — 

Library Selooco 
.47 .59 — 



3.7$ 
0,20 

3.70 
O.U 

3.52 
0.41 

3.60 
0.40 

3.68 
0.25 

'3. 58 
0.30 

3.60 
0.34 

3.73 
0.31 





. .33 


3.76 






0.27 




.46 


3.32 


ft 


• .77) 


0.32 




.13 


3.07 


ft 


• .63) 


0.32 



Not*: «oUioU corroUV'" cotfHclowts <» v«lu»« in oaronthos** 
ift gCPA coUon) art not raportad in aU 1n»tane*« oua ori- 
•arUy to sa«ela «1«a and/or «i»sfn« data contidarat<on«. 

•Croa«-v»lldatad cospoalea of pradtctora. 
••Ctt local (V*<J/2) ooly waa raportad. 

*Indlcatea degree orientation of students in the sample: M - master's. M&D 
both prospective master's aod doctoral students. 
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Su«Mr.y «« V«U4Uy CetffUUntt Ur QKI and 



SehooX (cotf«4) FrtUleeor bmh * •.4. 



Ccid OA 





School 221 
(19,19,11.14) 


569 

129 


302 
UO 


930 


n&D 


School 221 
(14.14.14.14.) 


4«7 

. lii 


446 

uo 


382 
102 


M&O 

VMS tr 


School 204 
(10.10.-.—) 


S38 
103 


471 
79 




n&O 


School 204 
( «. «.-.-) 


399 


339 




M&D 


School 293 
(37,37.-.-) 


571 
Ul 


S39 
U6 




M 


School 0(9 
(41. 41. 3S. 33) 


SOS 
101 


316* 
113 


307 
74 


M 


School 123 
(37.37.13.-) 


477 
.121 


4S2 
7$ 


512 
104 


M&O 


School 204 
(6«.64.— .— ) 


S41 
U2 


SIS 
147 




D 


School 033 
(.«, 9. 9,—) 


32 


700 
99 


771 
81 


M&O 


School 101 
(W.16,-,16) 


ft7S 
60 


644 
72 




M&O 


School US 
(24.23.-,24) 


693 
S9 


637 
86 






School 204 
(17,17, 9,-) 


639 
95 


349 
103 


691 
99 


IH- 


School 231 
•(1*,14,— .— ) 


634 
S3 


628 
136 






School 2U ' 
v3i.3*. J4.— ) 


. S21 
93 


483 
97 


317 

6a 



3.39 
Q,2t 

3.36 
0,49 



.30 



3.23 
0.37 



3.30 



3.66 
0.33 



.57 .M 

CorMtt 

.37 -.37 — — 



^ Sp«aish 

(♦)* (-)* — 



roiroisn L«as t Lie 
.23 .10 — 



Mtt«lc 

.31 .38 .18 
Music 

.U .09 .29 
Mtt«lc 

14 -.04 — 

Philosophy 
(•)• (-)* (-)* 

Ptillesophy 
.37 •.48 — 

Philosophy 
.29 .01 — 

Philosophy 
.42 .22 (♦)• 

Philosophy 
.26 .30 — 

Music 

.29 .02 .08 



.23 
<»• .33) 



.24 
.77 



.>.17 
0.27 

2.79 
0.38 

3.88 
0.29 

3.87 

o.a 



3. 39 
0.33 



. 3.70 
0.24 

3.6S 
0.32 

3.83 
0.34 

3.81 
0.22 

3.2S 
0.33 

3.44 
0.37 

3.71 
0.27 

2.93 
0.78 

J.3« 
0.28 



Noto: MuUiol* corr#Ut*on co»ff1ei»nt» (8 vtluos in 0«<>*ntn*$«s 
1ft UGP* coUnn) «rf not r»oort»d tn «U <n»tsn<»« duf or<- 
•«rtiy to ssmeU iUt snd/or misslns d«t« eons^Of rstiont. 

*Co«eeici«ne b«sod oa loos eh«a 10 casos; siga iadicaccs 41r«ci'ioa 
of rolacionshlp . 

^Indicates degree orientation of students in sample: M&D <" both prospective 
master's and doctoral students. M • master's students. D « doctoral students 
I>f « primarily prospective doctoral students with several master's students, 
M»> « primarily master's students with several prospective doctoral students. 
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Appendlx D 

Examination of Departmental S^ples with 
Deviant Weights for Predictor a 
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Appendlx D 

Examination of Dapartnental Samples With Deviant 
Weights for Predictors 



The test results sunnarized in Table 16 indicate that the data 
conform generally to the connon-weights hypothesis. Iioifever,'what 
about the deviant departments? What, characteristics in the data may 
be associated with the observed outcomes? To shed liflSht on these 
questions, a detailed examination was made of the data for all 
departments in which one or more .predictors was Identified .as having 
a slope differing significantly from the pooled estimate. 

Table D-1 shows zero-order validity coefficients for V, SRE-Q, 
and UGPA, as available, for 12 departmental samples in which some 
departure from. the common slopes hypothesis was indicated. Slopes 
for the variables that are specially marked were different from 
. pooled estimates in the V, Q, and/or the V, Q, UGPA analyses. It 
may be noted that of the 15 marked coefficients, seven are, positive 
arid eight are negative* In almost every instance, examination of the 
original data for the samples involved revealed certain conditions 
that help to account for either the tmusually high positive coefficients 
(and the correspondingly large positive regression weights) or the 
theoretically anomalous negative coefficients (and the corresponding 
negative regression weights). 

Detail regarding each sample is provided following a. jgeneral 
summary of the basic patterns of findings regarding the deviations: 

• Positive deviations . In four samples characterized by 
upusually high positive weights for GRE-V and/or GRE-Q, the 
observed result is associated with one pr more atypical data 
sets (In the comparatively small samples) for individuals who 
are in certain Identifiable ways "atypical"— i.e., members of 
minority groups with a very low test score and also very low 
graduate school grades, and/of foreign students for whom English 
may not be the native language* 

It 

In the sample of 11 students for Biology (B), for example, 
one student (foreign by inference from name) presented a GRE-V 
score of 320 (2.51 standard deviation below the mean) and earned 
a CPA of 3*07 (1*8A S*D**s below the mean)* Without this data 
set, the sample coefficient would have been *42 rather than *80* . 
The "inflated" GRE-V coefficient for Chemistry (F) is accounted 
for by one data set (foreign), while the unusually high GRE 
coefficients for Zoology (J) and Psychology (I) are heavily 
Influenced by data sets for two and eight minority students, 
respectively, with atypically low GRE scores and graduate GPA* 



departments are Identified by letter in the detailed descriptions 
which follow* Letters are as indicated In Table D-1. 
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Table D-1 



Validity Coefficients for Predictors In Departmental Sauries 
.with Deviant Weights on Designated Predictors 



Department (School) 



N 



Validity coefficient for predictor 



GRE^V 



GRB>Q 



UGPA 



Chemistry 

Chemistry 

* 


(A) 
CF) 


52 
13 


-.21* 
.40* 


.17 
.15 

1 • 


.37 
.03 


Biology 
Botany' 
Biology 
Zoology 


(B) 
(C) 
(D) 
(J) 


11 
15 
26 
19 


.80** 
.32 
.05 
.60 


-.03 
, -.43** 
-.07# 
.72## 


.16 
-.13 


Psychology 
Psychology 
Psychology 
Psychology 


(H) 
(E) 
(I) 
(G) 


20 
27 
51 
46 


-.,36** 
>.16*** 

.52* 

.44 


.19 
-.31*** 
.54tf# 
.21' 


.29 
.11 
.52** 


History 
History 


(A) 
(C) 


46 
25 


.29 
.08 


.22 
-.3^*** 


.53** 
-.01** 


English 


(None) 











Note: If the coefficient for a predictor is not specially marked, the 

corresponding weight was not identified as differing significantly 
from the pooled estimate. 

* 

Corresponding weight deviant in analysis with Q constant but not with" 
Q, U constant 

^ Corresponding weight deviant 
*** 

Corresponding weights deviant in analyses with V, and with V, CPA 

# c- 
Corresponding weight deviant with V. UGPA constant, but not with V only 

Corresponding weigtft deviant in analysis with V, but not witfi V, UQPA 
constant ^ 
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No specific sanple characteristics could be Identified 
to help "explain" the atyplcally high positive coefficients for 
UGFA In two samples* 

• Wegatlye deviations * In samples with anomalous negative 
coeffliilents, outcomes were associated with one or' more of the 
following conditions : (1> one or two extremely atypical data 
sets or outliers (Chemistry, (A), Botany (C)» Psychology (H), 
History (C) ] whlph heavily influenced results; (2) extremely 
skewed grade distribution Isee especially Psjrchology (E)l; 
(3) minority and/or foreign student (see Chemistry (A), 
History (C) ] ; or (A) confounding Interactions between level of 
GRE-scores, level of GPA» and year of entry (Biology (D)]. 

Detailed Description of Departmental Data 

It Is helpful first to consider the samples In which an atyplcally large 
positive weighting was present and then those Xn which large negative 
weighting was present for one or more predictors* 

Positive Deviations 

Chemistry (F) : Sample of N - 11 Includes two students, who by name, 
identification appear to be foreign,. with extremely low GRE-V score? and 
quite, low graduate CPA (V « 290, GPA « 2,04 on 4,00 scale; V » 310, 
CPA - 3,18), CKi.-Q score is in average range. 

Biology (B) : Sample of 11 students includes one foreign student 
(by name ID) with very low GRE-V (320) and very low GPA (3,07), 

Zoology (J): Sample N » 19. Two minority students in sample, with 
data as follows: 



(Student 1) 



V 


Q 


UGPA 


Graduate 


370 


420 


3.61 


2.75 


430 


410 

t 


3.04 


2,57 


564 


621 


3.28 


3.65 


96 


94 


0.35 


0.40 



Dept. mean 
Dept. slgma 



High positive weights for V and Q are due primarily to positive covariation 
contributed by these two cases. Negative weight for UGPA (not Identified as 
significantly deviant) Is accounted for primarily Student 1. 



\ 
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Poatttve Devli^tlons (cont.) * 

* , 

Psychology (I)t As for Zoology in School J» the unusually high 
positive coefficients (weights) for V and Q ifere due to the fact that 
minority students with low GRE scores also tended to earn low graduati 
grades (i.e., correlated means)* / • 



Psychology (G): 

Psychology (A): No specific patterns such as those outlined above 
can be identified to account for the high positive contribution of the UGPA 
'in these two samples. 

Negative Deviations ^ 

Chemistry (A): Correlation of GRE-V with Graduate CPA is -.21 in- this 
sample (N ** 52). One of three minority students (Oriental-American) haa 
a GRE-V score of 310 (-4.21 standard deviations) and a Graduate GPA t)f 
"A" (+2.84 standard deviation^). Elimination of this data set yields a 
sample in which the coefficient for GRE-V is .02; This individual's Q 
score was 760 (+1.00 S.D. , approximately). 

Botany (C) : GRE-Q was negatively weighted in this sample (N - 15, 

Mean Q « 591, S.D. « 89). IVo students with highest Q score, both 720, had , 
the lowest and second lowest Graduate GPA (3.10 and 3.20) in group with 
Mean. GPA « 3.70, S.D. - 0.30. No ethnic or language data were coded for this 
sample. * 

Psychology (H): Negative weighting was observed for GRE-V in thiis 
sample of 20 cases. Mean V - '621, S-D. » 60; mean Graduate GPA » 3.70, 
S.D. - 0.24. Individual with lowest Graduate pPA (3.19, or -2.13 S.D.) 
was one of two individuals with highest V score (740, or +1.98 S.D.), The 
other student with V « 740 earned GPA somewhat below average. Both these 
students had below average Q scores. No ethnic or language data were coded 
for this sample. 

P.sychology (E): Coafficients for V and Q were negative in this saiq>le 

ot J 7 cases: 



GRE-Verbal 



Graduate GPA 
Less 4 . 00 

than 4.00 



600+ 



5 



5 



Less than 600 



5 



12 



Total 



10 



17 



Neif^^tiv Devlatlona (cont.) 

Tht Graduate CPA distribution for this departnent was heavily skaired 
Mtatively: 



<A average) A. 00 17 
r 3.80 6 

3.70 1 Mean - 3.89 
3.60 1 ^ S.D. ,-0.16 
3.50 2 

It seems probable that differences among students are not reliably measuried. 

History (C): In this department of 21 cases, Q and UGPA were X ' 
identified as deviant (negative zero-order correlation with Graduate GPA) . 
Student with lowest Q (310, -2.22 S.D.) earned 4.00 Graduate GPA (+1.25 S.D/); 
student vith highest Q (650/ + 1.69 S.D.) had lowest Graduate GPA' 
(2.80, 1.75 S.D.). Sample heterogeneity involved— several foreign students. 

Biol(^gy (D): Negative weight for Q. in this departmental sample (N - 26): 

GRE-Q GPA distribution 

Below 3.25- 3.7^ 
3.25 3.74 . 

700+ 4* 2 3 

600-699 2 7-^2 

Below 600 14 1 



^Cases in this cell account for negative correlation. 



V- 



In this department, students entering in 1975 had lower ORE scores than those 
entering in 1974 but higher mean GPA. Relationship among year of entry, 
GRE variables and Graduate GPA was as shown below: 



GRE-V 
GRE-Q 



1975 vs. 1974 Graduate GPA 

-.37 (*75 lower) 'OS 
-.24 (*75 lower) -.07 



Grad GPA .18 ('75 higher) 



Some Implications of the Flndln'^8 



These flndl.ng8 point up the Impact of one pr tw aberrational data* sets, 
or outliers, on tMe magnitude and the signs of validity coefficients In small 
samples. The negative coefficients,- of course, are anomalous— >l.e. , coefflQlents 
reflecting the relationship between GRE and UGPA predictors, on the one hand*; 
and first-year Graduate CPA, on the other, should be positive, a priori. Given 
the potential for anomalous "outlier" Impact in small samples, the over- 
whelmingly positive distribution of coefficients obtained in the Cooperative 
Studies for GRE and. UGPA in departmental samples with very small Ns, on the ^ 
average. Indicates a remarkable degree of underlying "regularity" in such data. 
Careful attention to saoq>le definition clearly Is important. 
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